2024-09-20T19:25:47.864 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-20T19:25:47.895 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-20T19:25:47.987 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi/7912799 branch: wip-adk-testing-2024-09-20-0910 description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: adking@redhat.com first_in_suite: false job_id: '7912799' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/squid ' name: adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-09-20-0910 ceph: cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: global: mon pg warn min per osd: 0 mgr: client mount timeout: 30 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - filesystem is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - filesystem is offline - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-adk-testing-2024-09-20-0910 sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 8031 sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 sleep_before_teardown: 0 subset: 1/20 suite: orch:cephadm suite_branch: wip-adk-testing-2024-09-20-0910 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 targets: smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO2C+DS9gGIq3b/wo8kU6gxnA7EFu+UdHUFSXi045NoRxzpsfdkpb6kxaApx3WPcMHaz6omzaEubaX2NX5Du5zA= smithi160.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI7aQV0Ig/zKJ4gOQvmcQKSXPl/NumepdwaxgQiF4iEkw8ORY3gHN2bLo7qwBv3S44sP0QlohvBKPEenjH7n5Y4= tasks: - install: branch: squid exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: compiled_cephadm_branch: squid conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:squid roleless: true - print: '**** done end installing squid cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-20_18:25:04 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-09-20T19:25:47.987 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa; will attempt to use it 2024-09-20T19:25:47.988 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks 2024-09-20T19:25:47.988 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-20T19:25:47.990 INFO:teuthology.task.internal:Checking packages... 2024-09-20T19:25:48.011 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e8baca1eac6c3c9c88bf2ea4493e448bd54936a7' 2024-09-20T19:25:48.011 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-20T19:25:48.011 INFO:teuthology.packaging:ref: None 2024-09-20T19:25:48.011 INFO:teuthology.packaging:tag: None 2024-09-20T19:25:48.011 INFO:teuthology.packaging:branch: wip-adk-testing-2024-09-20-0910 2024-09-20T19:25:48.011 INFO:teuthology.packaging:sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:25:48.011 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-09-20-0910 2024-09-20T19:25:48.155 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5127.ge8baca1e 2024-09-20T19:25:48.157 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-20T19:25:48.162 INFO:teuthology.task.internal:no buildpackages task found 2024-09-20T19:25:48.163 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-20T19:25:48.210 INFO:teuthology.task.internal:Saving configuration 2024-09-20T19:25:48.231 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-20T19:25:48.237 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-20T19:25:48.263 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi/7912799', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 19:20:39.545971', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO2C+DS9gGIq3b/wo8kU6gxnA7EFu+UdHUFSXi045NoRxzpsfdkpb6kxaApx3WPcMHaz6omzaEubaX2NX5Du5zA='} 2024-09-20T19:25:48.286 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi/7912799', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 19:20:39.547906', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI7aQV0Ig/zKJ4gOQvmcQKSXPl/NumepdwaxgQiF4iEkw8ORY3gHN2bLo7qwBv3S44sP0QlohvBKPEenjH7n5Y4='} 2024-09-20T19:25:48.286 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-20T19:25:48.292 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-09-20T19:25:48.292 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-09-20T19:25:48.292 INFO:teuthology.run_tasks:Running task console_log... 2024-09-20T19:25:48.351 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7faad8b08280>, signals=[15]) 2024-09-20T19:25:48.351 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-20T19:25:48.357 INFO:teuthology.task.internal:Opening connections... 2024-09-20T19:25:48.357 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-09-20T19:25:48.358 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:25:48.437 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi160.front.sepia.ceph.com 2024-09-20T19:25:48.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:25:48.515 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-20T19:25:48.520 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-09-20T19:25:48.539 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-09-20T19:25:48.539 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-09-20T19:25:48.597 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-09-20T19:25:48.597 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-09-20T19:25:48.597 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-09-20T19:25:48.597 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-09-20T19:25:48.597 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-09-20T19:25:48.598 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-09-20T19:25:48.598 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-20T19:25:48.598 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-09-20T19:25:48.598 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-09-20T19:25:48.598 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-20T19:25:48.598 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-09-20T19:25:48.598 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-20T19:25:48.598 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-20T19:25:48.598 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-20T19:25:48.599 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-09-20T19:25:48.630 DEBUG:teuthology.orchestra.run.smithi160:> uname -m 2024-09-20T19:25:48.649 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2024-09-20T19:25:48.649 DEBUG:teuthology.orchestra.run.smithi160:> cat /etc/os-release 2024-09-20T19:25:48.707 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="CentOS Stream" 2024-09-20T19:25:48.708 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="9" 2024-09-20T19:25:48.708 INFO:teuthology.orchestra.run.smithi160.stdout:ID="centos" 2024-09-20T19:25:48.708 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE="rhel fedora" 2024-09-20T19:25:48.708 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="9" 2024-09-20T19:25:48.708 INFO:teuthology.orchestra.run.smithi160.stdout:PLATFORM_ID="platform:el9" 2024-09-20T19:25:48.708 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-20T19:25:48.708 INFO:teuthology.orchestra.run.smithi160.stdout:ANSI_COLOR="0;31" 2024-09-20T19:25:48.708 INFO:teuthology.orchestra.run.smithi160.stdout:LOGO="fedora-logo-icon" 2024-09-20T19:25:48.708 INFO:teuthology.orchestra.run.smithi160.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-20T19:25:48.708 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://centos.org/" 2024-09-20T19:25:48.708 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-20T19:25:48.708 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-20T19:25:48.709 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-20T19:25:48.709 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2024-09-20T19:25:48.739 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-20T19:25:48.751 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-20T19:25:48.757 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-20T19:25:48.757 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-09-20T19:25:48.759 DEBUG:teuthology.orchestra.run.smithi160:> test '!' -e /home/ubuntu/cephtest 2024-09-20T19:25:48.776 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-20T19:25:48.788 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-20T19:25:48.788 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-09-20T19:25:48.817 DEBUG:teuthology.orchestra.run.smithi160:> test -z $(ls -A /var/lib/ceph) 2024-09-20T19:25:48.904 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-20T19:25:48.953 INFO:teuthology.run_tasks:Running task kernel... 2024-09-20T19:25:48.975 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-20T19:25:48.975 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-20T19:25:48.975 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T19:25:48.976 DEBUG:teuthology.orchestra.run.smithi160:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T19:25:48.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:48.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:25:48.994 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2024-09-20T19:25:48.995 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-09-20T19:25:49.052 INFO:teuthology.orchestra.run.smithi160.stdout:5.14.0-503.el9.x86_64 2024-09-20T19:25:49.052 INFO:teuthology.task.kernel:Running kernel on smithi160: 5.14.0-503.el9.x86_64 2024-09-20T19:25:49.052 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum install -y kernel 2024-09-20T19:25:49.053 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-503.el9.x86_64 2024-09-20T19:25:49.053 INFO:teuthology.task.kernel:Running kernel on smithi114: 5.14.0-503.el9.x86_64 2024-09-20T19:25:49.053 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2024-09-20T19:25:52.044 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.3 MB 00:01 2024-09-20T19:25:53.455 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.3 MB 00:02 2024-09-20T19:25:54.894 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 12 MB/s | 20 MB 00:01 2024-09-20T19:25:55.397 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-20T19:25:59.370 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-20T19:26:00.647 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.5 MB 00:01 2024-09-20T19:26:01.612 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-20T19:26:02.400 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-20T19:26:05.253 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - Extras packages 5.6 kB/s | 19 kB 00:03 2024-09-20T19:26:06.428 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-20T19:26:07.956 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-20T19:26:09.854 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-20T19:26:09.854 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-20T19:26:09.948 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:26:09.953 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:26:09.953 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-20T19:26:09.953 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:26:09.953 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:26:09.953 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-20T19:26:09.953 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-20T19:26:09.953 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-20T19:26:09.953 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-20T19:26:09.954 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-20T19:26:09.954 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:26:09.954 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:26:09.954 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:26:09.954 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-09-20T19:26:09.954 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:26:09.954 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 79 M 2024-09-20T19:26:09.955 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 126 M 2024-09-20T19:26:09.955 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:26:10.316 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 338 kB/s | 36 kB 00:00 2024-09-20T19:26:10.699 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-09-20T19:26:11.083 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 37 MB/s | 29 MB 00:00 2024-09-20T19:26:11.391 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 29 MB/s | 35 MB 00:01 2024-09-20T19:26:11.392 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:26:11.392 INFO:teuthology.orchestra.run.smithi114.stdout:Total 55 MB/s | 79 MB 00:01 2024-09-20T19:26:11.884 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-20T19:26:12.074 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:26:12.404 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:26:12.404 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:26:13.366 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:26:13.367 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:26:13.824 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-20T19:26:13.824 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-20T19:26:13.903 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:26:13.908 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:26:13.908 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-09-20T19:26:13.908 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:26:13.908 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:26:13.908 INFO:teuthology.orchestra.run.smithi160.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-20T19:26:13.908 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-20T19:26:13.908 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-20T19:26:13.908 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-20T19:26:13.908 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-20T19:26:13.908 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:26:13.908 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:26:13.908 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:26:13.908 INFO:teuthology.orchestra.run.smithi160.stdout:Install 4 Packages 2024-09-20T19:26:13.909 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:26:13.909 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 79 M 2024-09-20T19:26:13.909 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 126 M 2024-09-20T19:26:13.909 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:26:14.269 INFO:teuthology.orchestra.run.smithi160.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 317 kB/s | 36 kB 00:00 2024-09-20T19:26:14.869 INFO:teuthology.orchestra.run.smithi160.stdout:(2/4): kernel-modules-core-5.14.0-508.el9.x86_6 48 MB/s | 29 MB 00:00 2024-09-20T19:26:15.327 INFO:teuthology.orchestra.run.smithi160.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-09-20T19:26:15.761 INFO:teuthology.orchestra.run.smithi160.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 22 MB/s | 35 MB 00:01 2024-09-20T19:26:15.761 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:26:15.761 INFO:teuthology.orchestra.run.smithi160.stdout:Total 43 MB/s | 79 MB 00:01 2024-09-20T19:26:15.776 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:26:16.416 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-20T19:26:16.442 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:26:16.500 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T19:26:16.785 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:26:16.785 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:26:17.738 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:26:17.738 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:26:19.245 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T19:26:19.368 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T19:26:20.205 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:26:20.847 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-20T19:26:20.928 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T19:26:23.713 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T19:26:23.839 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T19:26:24.186 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T19:26:24.294 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:26:28.670 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T19:26:28.706 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:26:29.136 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:26:33.557 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:27:07.229 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:27:07.268 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:27:07.592 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:27:07.592 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-20T19:27:07.592 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T19:27:07.593 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T19:27:08.523 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:27:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:27:08.524 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:27:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-20T19:27:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-20T19:27:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-20T19:27:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-20T19:27:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:27:08.524 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:27:08.876 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T19:27:08.967 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T19:27:08.967 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T19:27:08.967 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-20T19:27:08.967 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T19:27:08.967 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T19:27:08.968 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-09-20T19:27:08.968 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T19:27:08.968 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T19:27:09.026 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T19:27:09.026 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T19:27:09.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-09-20T19:27:09.058 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-09-20T19:27:09.058 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-09-20T19:27:09.058 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-09-20T19:27:09.058 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 82.el9 2024-09-20T19:27:09.058 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-09-20T19:27:09.059 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-20T19:27:09.059 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-09-20T19:27:09.059 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8274670 2024-09-20T19:27:09.059 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-09-20T19:27:09.059 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-20T19:27:09.059 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-20T19:27:09.059 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-20T19:27:09.059 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-20T19:27:09.059 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-09-20T19:27:09.059 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-09-20T19:27:09.059 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-20T19:27:09.060 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-09-20T19:27:09.060 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-09-20T19:27:09.060 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:27:09.060 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-20T19:27:09.060 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-20T19:27:09.060 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-20T19:27:09.060 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-09-20T19:27:09.060 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:27:09.060 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-09-20T19:27:09.061 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-20T19:27:09.061 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 5.14.0-508.el9.x86_64 2024-09-20T19:27:09.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-20T19:27:09.689 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-09-20T19:27:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-20T19:27:11.068 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-09-20T19:27:11.071 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-09-20T19:27:11.086 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.NZOqxvEyZq 2024-09-20T19:27:11.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NZOqxvEyZq 2024-09-20T19:27:11.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.NZOqxvEyZq 2024-09-20T19:27:11.302 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.NZOqxvEyZq is 6KB 2024-09-20T19:27:11.350 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.NZOqxvEyZq 2024-09-20T19:27:11.365 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-09-20T19:27:11.422 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:27:11.428 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-20T19:27:11.428 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-20T19:27:11.428 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-20T19:27:11.428 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-20T19:27:11.429 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-20T19:27:11.462 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:27:11.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-09-20T19:27:11.776 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:27:11.776 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-20T19:27:11.776 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T19:27:11.776 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T19:27:12.621 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T19:27:12.621 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:27:12.621 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:27:12.621 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-20T19:27:12.622 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-20T19:27:12.622 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-20T19:27:12.622 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-20T19:27:12.622 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:27:12.622 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:27:12.968 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T19:27:13.056 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T19:27:13.056 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T19:27:13.057 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-20T19:27:13.057 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T19:27:13.057 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T19:27:13.057 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-09-20T19:27:13.057 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi160.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T19:27:13.057 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T19:27:13.115 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T19:27:13.116 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T19:27:13.116 DEBUG:teuthology.orchestra.run.smithi160:> sudo rpm -qi grub2-tools 2024-09-20T19:27:13.147 INFO:teuthology.orchestra.run.smithi160.stdout:Name : grub2-tools 2024-09-20T19:27:13.148 INFO:teuthology.orchestra.run.smithi160.stdout:Epoch : 1 2024-09-20T19:27:13.148 INFO:teuthology.orchestra.run.smithi160.stdout:Version : 2.06 2024-09-20T19:27:13.148 INFO:teuthology.orchestra.run.smithi160.stdout:Release : 82.el9 2024-09-20T19:27:13.148 INFO:teuthology.orchestra.run.smithi160.stdout:Architecture: x86_64 2024-09-20T19:27:13.148 INFO:teuthology.orchestra.run.smithi160.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-20T19:27:13.148 INFO:teuthology.orchestra.run.smithi160.stdout:Group : Unspecified 2024-09-20T19:27:13.148 INFO:teuthology.orchestra.run.smithi160.stdout:Size : 8274670 2024-09-20T19:27:13.148 INFO:teuthology.orchestra.run.smithi160.stdout:License : GPLv3+ 2024-09-20T19:27:13.148 INFO:teuthology.orchestra.run.smithi160.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-20T19:27:13.148 INFO:teuthology.orchestra.run.smithi160.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-20T19:27:13.148 INFO:teuthology.orchestra.run.smithi160.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-20T19:27:13.148 INFO:teuthology.orchestra.run.smithi160.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-20T19:27:13.148 INFO:teuthology.orchestra.run.smithi160.stdout:Packager : builder@centos.org 2024-09-20T19:27:13.149 INFO:teuthology.orchestra.run.smithi160.stdout:Vendor : CentOS 2024-09-20T19:27:13.149 INFO:teuthology.orchestra.run.smithi160.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-20T19:27:13.149 INFO:teuthology.orchestra.run.smithi160.stdout:Summary : Support tools for GRUB. 2024-09-20T19:27:13.149 INFO:teuthology.orchestra.run.smithi160.stdout:Description : 2024-09-20T19:27:13.149 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:27:13.149 INFO:teuthology.orchestra.run.smithi160.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-20T19:27:13.149 INFO:teuthology.orchestra.run.smithi160.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-20T19:27:13.149 INFO:teuthology.orchestra.run.smithi160.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-20T19:27:13.149 INFO:teuthology.orchestra.run.smithi160.stdout:hardware devices. 2024-09-20T19:27:13.149 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:27:13.149 INFO:teuthology.orchestra.run.smithi160.stdout:This subpackage provides tools for support of all platforms. 2024-09-20T19:27:13.150 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-20T19:27:13.150 INFO:teuthology.task.kernel:Updating grub on smithi160 to boot 5.14.0-508.el9.x86_64 2024-09-20T19:27:13.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-20T19:27:13.805 INFO:teuthology.orchestra.run.smithi160.stderr:Generating grub configuration file ... 2024-09-20T19:27:15.130 INFO:teuthology.orchestra.run.smithi160.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-20T19:27:15.155 INFO:teuthology.orchestra.run.smithi160.stderr:done 2024-09-20T19:27:15.157 DEBUG:teuthology.orchestra.run.smithi160:> mktemp 2024-09-20T19:27:15.173 INFO:teuthology.orchestra.run.smithi160.stdout:/tmp/tmp.JGbba66K2B 2024-09-20T19:27:15.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.JGbba66K2B 2024-09-20T19:27:15.241 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0666 /tmp/tmp.JGbba66K2B 2024-09-20T19:27:15.376 DEBUG:teuthology.orchestra.remote:smithi160:/tmp/tmp.JGbba66K2B is 6KB 2024-09-20T19:27:15.396 DEBUG:teuthology.orchestra.run.smithi160:> rm -fr /tmp/tmp.JGbba66K2B 2024-09-20T19:27:15.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo /bin/ls /boot/loader/entries || true 2024-09-20T19:27:15.480 INFO:teuthology.orchestra.run.smithi160.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-20T19:27:15.480 INFO:teuthology.orchestra.run.smithi160.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-20T19:27:15.480 INFO:teuthology.orchestra.run.smithi160.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-20T19:27:15.480 INFO:teuthology.orchestra.run.smithi160.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-20T19:27:15.481 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-20T19:27:15.649 DEBUG:teuthology.orchestra.run.smithi160:> sudo shutdown -r now 2024-09-20T19:27:41.598 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-09-20T19:27:41.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-20T19:27:41.606 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:27:45.654 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-09-20T19:27:45.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T19:27:45.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:28:04.079 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T19:28:13.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T19:28:13.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:28:16.111 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-09-20T19:28:16.146 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T19:28:25.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-20T19:28:25.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:28:28.149 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T19:28:28.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:28:43.503 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-09-20T19:28:46.575 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T19:28:55.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-20T19:28:55.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:28:55.876 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-09-20T19:28:56.295 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-20T19:28:56.295 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-508.el9.x86_64"... 2024-09-20T19:28:56.295 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-09-20T19:28:56.309 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-508.el9.x86_64 2024-09-20T19:28:56.309 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-20T19:28:56.309 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T19:28:56.310 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-20T19:28:57.311 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-09-20T19:28:57.311 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T19:28:57.406 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2024-09-20T19:28:57.439 DEBUG:teuthology.parallel:result is None 2024-09-20T19:29:01.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T19:29:01.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:29:01.882 DEBUG:teuthology.orchestra.run.smithi160:> true 2024-09-20T19:29:02.427 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T19:29:02.428 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-508.el9.x86_64"... 2024-09-20T19:29:02.428 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2024-09-20T19:29:02.442 INFO:teuthology.orchestra.run.smithi160.stdout:5.14.0-508.el9.x86_64 2024-09-20T19:29:02.442 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-20T19:29:02.442 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T19:29:02.442 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-20T19:29:03.444 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-09-20T19:29:03.444 DEBUG:teuthology.orchestra.run.smithi160:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T19:29:03.577 INFO:teuthology.orchestra.run.smithi160.stdout:ttyS1 2024-09-20T19:29:03.600 DEBUG:teuthology.parallel:result is None 2024-09-20T19:29:03.600 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-20T19:29:03.612 INFO:teuthology.task.internal:Creating test directory... 2024-09-20T19:29:03.612 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T19:29:03.614 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T19:29:03.631 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-20T19:29:03.637 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-20T19:29:03.643 INFO:teuthology.task.internal:Creating archive directory... 2024-09-20T19:29:03.643 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T19:29:03.670 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T19:29:03.763 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-20T19:29:03.768 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-20T19:29:03.769 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T19:29:03.771 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T19:29:03.799 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T19:29:03.799 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T19:29:03.809 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T19:29:03.811 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T19:29:03.812 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-20T19:29:03.819 INFO:teuthology.task.internal:Configuring sudo... 2024-09-20T19:29:03.819 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T19:29:03.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T19:29:03.880 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-20T19:29:03.888 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-20T19:29:03.888 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T19:29:03.921 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T19:29:03.936 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T19:29:04.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T19:29:04.065 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T19:29:04.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T19:29:04.188 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:29:04.188 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T19:29:04.253 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T19:29:04.294 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T19:29:04.350 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T19:29:04.434 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T19:29:04.472 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-20T19:29:04.473 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T19:29:04.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-09-20T19:29:04.559 DEBUG:teuthology.orchestra.run.smithi160:> sudo service rsyslog restart 2024-09-20T19:29:04.622 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T19:29:04.641 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T19:29:05.151 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-20T19:29:05.160 INFO:teuthology.task.internal:Starting timer... 2024-09-20T19:29:05.160 INFO:teuthology.run_tasks:Running task pcp... 2024-09-20T19:29:05.197 INFO:teuthology.run_tasks:Running task selinux... 2024-09-20T19:29:05.205 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-20T19:29:05.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-09-20T19:29:05.272 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-09-20T19:29:05.274 DEBUG:teuthology.orchestra.run.smithi160:> sudo service auditd rotate 2024-09-20T19:29:05.339 INFO:teuthology.orchestra.run.smithi160.stdout:Rotating logs: 2024-09-20T19:29:05.341 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-20T19:29:05.341 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-09-20T19:29:05.373 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-09-20T19:29:05.373 DEBUG:teuthology.orchestra.run.smithi160:> /usr/sbin/getenforce 2024-09-20T19:29:05.402 INFO:teuthology.orchestra.run.smithi160.stdout:Permissive 2024-09-20T19:29:05.402 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi114.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi160.front.sepia.ceph.com': 'permissive'} 2024-09-20T19:29:05.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-20T19:29:05.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:05.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-20T19:29:05.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:29:05.467 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-20T19:29:05.467 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-09-20T19:29:05.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo /usr/sbin/setenforce permissive 2024-09-20T19:29:05.547 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-20T19:29:05.557 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-20T19:29:05.559 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-20T19:29:05.668 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-20T19:29:05.677 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-20T19:29:05.678 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi114.front.sepia.ceph.com,smithi160.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-20T19:32:46.859 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi114.front.sepia.ceph.com'), Remote(name='ubuntu@smithi160.front.sepia.ceph.com')] 2024-09-20T19:32:46.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-20T19:32:46.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:32:46.933 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-09-20T19:32:47.009 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-20T19:32:47.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T19:32:47.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T19:32:47.083 DEBUG:teuthology.orchestra.run.smithi160:> true 2024-09-20T19:32:47.163 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T19:32:47.163 INFO:teuthology.run_tasks:Running task clock... 2024-09-20T19:32:47.172 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-20T19:32:47.173 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T19:32:47.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T19:32:47.175 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T19:32:47.175 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T19:32:47.201 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-20T19:32:47.215 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-20T19:32:47.232 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-20T19:32:47.239 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-09-20T19:32:47.247 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-20T19:32:47.250 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-09-20T19:32:47.264 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-20T19:32:47.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: ntpd: command not found 2024-09-20T19:32:47.278 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-20T19:32:47.282 INFO:teuthology.orchestra.run.smithi160.stdout:506 Cannot talk to daemon 2024-09-20T19:32:47.296 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-20T19:32:47.310 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-20T19:32:47.337 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-09-20T19:32:47.341 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T19:32:47.341 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-09-20T19:32:47.341 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:32:47.341 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:32:47.341 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:32:47.341 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:32:47.372 INFO:teuthology.orchestra.run.smithi160.stderr:bash: line 1: ntpq: command not found 2024-09-20T19:32:47.375 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T19:32:47.375 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2024-09-20T19:32:47.376 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:32:47.376 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:32:47.376 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:32:47.376 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T19:32:47.377 INFO:teuthology.run_tasks:Running task install... 2024-09-20T19:32:47.386 DEBUG:teuthology.task.install:project ceph 2024-09-20T19:32:47.386 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e8baca1eac6c3c9c88bf2ea4493e448bd54936a7'}} 2024-09-20T19:32:47.386 DEBUG:teuthology.task.install:config {'branch': 'squid', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'e8baca1eac6c3c9c88bf2ea4493e448bd54936a7'} 2024-09-20T19:32:47.387 INFO:teuthology.task.install:Using flavor: default 2024-09-20T19:32:47.392 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-20T19:32:47.392 INFO:teuthology.task.install:extra packages: [] 2024-09-20T19:32:47.392 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8baca1eac6c3c9c88bf2ea4493e448bd54936a7', 'tag': None, 'wait_for_package': False} 2024-09-20T19:32:47.392 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-20T19:32:47.392 INFO:teuthology.packaging:ref: None 2024-09-20T19:32:47.392 INFO:teuthology.packaging:tag: None 2024-09-20T19:32:47.392 INFO:teuthology.packaging:branch: squid 2024-09-20T19:32:47.393 INFO:teuthology.packaging:sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:32:47.393 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-20T19:32:47.394 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8baca1eac6c3c9c88bf2ea4493e448bd54936a7', 'tag': None, 'wait_for_package': False} 2024-09-20T19:32:47.394 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-20T19:32:47.394 INFO:teuthology.packaging:ref: None 2024-09-20T19:32:47.395 INFO:teuthology.packaging:tag: None 2024-09-20T19:32:47.395 INFO:teuthology.packaging:branch: squid 2024-09-20T19:32:47.395 INFO:teuthology.packaging:sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:32:47.395 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-20T19:32:47.621 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/ed8ee0f26ef720e6504cefa8be1a2d0872192de7/centos/9/flavors/default/ 2024-09-20T19:32:47.622 INFO:teuthology.task.install.rpm:Package version is 19.1.1-369.ged8ee0f2 2024-09-20T19:32:47.655 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/ed8ee0f26ef720e6504cefa8be1a2d0872192de7/centos/9/flavors/default/ 2024-09-20T19:32:47.655 INFO:teuthology.task.install.rpm:Package version is 19.1.1-369.ged8ee0f2 2024-09-20T19:32:47.734 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/ed8ee0f26ef720e6504cefa8be1a2d0872192de7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/ed8ee0f26ef720e6504cefa8be1a2d0872192de7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/ed8ee0f26ef720e6504cefa8be1a2d0872192de7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-20T19:32:47.734 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-20T19:32:47.735 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-20T19:32:47.761 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-20T19:32:47.761 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-20T19:32:47.761 INFO:teuthology.packaging:ref: None 2024-09-20T19:32:47.761 INFO:teuthology.packaging:tag: None 2024-09-20T19:32:47.762 INFO:teuthology.packaging:branch: squid 2024-09-20T19:32:47.762 INFO:teuthology.packaging:sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:32:47.762 DEBUG:teuthology.orchestra.run.smithi160:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-20T19:32:47.767 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/ed8ee0f26ef720e6504cefa8be1a2d0872192de7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/ed8ee0f26ef720e6504cefa8be1a2d0872192de7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/ed8ee0f26ef720e6504cefa8be1a2d0872192de7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-20T19:32:47.767 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:32:47.767 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-20T19:32:47.794 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-20T19:32:47.794 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-20T19:32:47.794 INFO:teuthology.packaging:ref: None 2024-09-20T19:32:47.794 INFO:teuthology.packaging:tag: None 2024-09-20T19:32:47.794 INFO:teuthology.packaging:branch: squid 2024-09-20T19:32:47.795 INFO:teuthology.packaging:sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:32:47.795 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-20T19:32:47.828 DEBUG:teuthology.orchestra.run.smithi160:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-20T19:32:47.860 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-20T19:32:47.903 DEBUG:teuthology.orchestra.run.smithi160:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-20T19:32:47.935 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-20T19:32:47.967 INFO:teuthology.orchestra.run.smithi160.stdout:check_obsoletes = 1 2024-09-20T19:32:47.969 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum clean all 2024-09-20T19:32:47.999 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-09-20T19:32:48.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-09-20T19:32:48.199 INFO:teuthology.orchestra.run.smithi160.stdout:45 files removed 2024-09-20T19:32:48.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-radosgw 2024-09-20T19:32:48.236 INFO:teuthology.orchestra.run.smithi114.stdout:45 files removed 2024-09-20T19:32:48.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw 2024-09-20T19:32:48.793 INFO:teuthology.orchestra.run.smithi160.stdout:ceph packages for x86_64 298 kB/s | 84 kB 00:00 2024-09-20T19:32:48.817 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 315 kB/s | 84 kB 00:00 2024-09-20T19:32:49.085 INFO:teuthology.orchestra.run.smithi160.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-09-20T19:32:49.301 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 26 kB/s | 12 kB 00:00 2024-09-20T19:32:49.361 INFO:teuthology.orchestra.run.smithi160.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-20T19:32:49.593 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 7.4 kB/s | 1.9 kB 00:00 2024-09-20T19:32:51.285 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.3 MB 00:01 2024-09-20T19:32:51.403 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.3 MB 00:02 2024-09-20T19:32:53.028 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-20T19:32:53.098 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - AppStream 33 MB/s | 20 MB 00:00 2024-09-20T19:32:57.388 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-20T19:32:58.595 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-20T19:32:59.658 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 19 kB 00:00 2024-09-20T19:33:00.543 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-20T19:33:02.734 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 6.7 kB/s | 19 kB 00:02 2024-09-20T19:33:03.685 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-20T19:33:06.024 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-20T19:33:08.259 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:33:08.278 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:33:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-20T19:33:08.278 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:33:08.278 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:33:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 11 M 2024-09-20T19:33:08.278 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrading: 2024-09-20T19:33:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: librados2 x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 3.4 M 2024-09-20T19:33:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1 x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 3.2 M 2024-09-20T19:33:08.278 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-20T19:33:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-20T19:33:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 5.4 M 2024-09-20T19:33:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 21 M 2024-09-20T19:33:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 25 k 2024-09-20T19:33:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-20T19:33:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-20T19:33:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-20T19:33:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2 x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 751 k 2024-09-20T19:33:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-20T19:33:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-20T19:33:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-20T19:33:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1 x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 506 k 2024-09-20T19:33:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-20T19:33:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2 x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 5.3 M 2024-09-20T19:33:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-20T19:33:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-20T19:33:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-20T19:33:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 45 k 2024-09-20T19:33:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 136 k 2024-09-20T19:33:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 162 k 2024-09-20T19:33:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-20T19:33:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 320 k 2024-09-20T19:33:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 301 k 2024-09-20T19:33:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 99 k 2024-09-20T19:33:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-20T19:33:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-20T19:33:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-20T19:33:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-20T19:33:08.281 INFO:teuthology.orchestra.run.smithi160.stdout:Installing weak dependencies: 2024-09-20T19:33:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-20T19:33:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:08.281 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:33:08.281 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:33:08.281 INFO:teuthology.orchestra.run.smithi160.stdout:Install 30 Packages 2024-09-20T19:33:08.281 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrade 2 Packages 2024-09-20T19:33:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:08.281 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 60 M 2024-09-20T19:33:08.282 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:33:09.066 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-09-20T19:33:09.075 INFO:teuthology.orchestra.run.smithi160.stdout:(1/32): ceph-base-19.1.1-369.ged8ee0f2.el9.x86_ 9.9 MB/s | 5.4 MB 00:00 2024-09-20T19:33:09.126 INFO:teuthology.orchestra.run.smithi160.stdout:(2/32): ceph-selinux-19.1.1-369.ged8ee0f2.el9.x 504 kB/s | 25 kB 00:00 2024-09-20T19:33:09.209 INFO:teuthology.orchestra.run.smithi160.stdout:(3/32): libcephfs2-19.1.1-369.ged8ee0f2.el9.x86 8.8 MB/s | 751 kB 00:00 2024-09-20T19:33:09.359 INFO:teuthology.orchestra.run.smithi160.stdout:(4/32): ceph-radosgw-19.1.1-369.ged8ee0f2.el9.x 13 MB/s | 11 MB 00:00 2024-09-20T19:33:09.384 INFO:teuthology.orchestra.run.smithi160.stdout:(5/32): libradosstriper1-19.1.1-369.ged8ee0f2.e 2.8 MB/s | 506 kB 00:00 2024-09-20T19:33:09.451 INFO:teuthology.orchestra.run.smithi160.stdout:(6/32): python3-ceph-argparse-19.1.1-369.ged8ee 675 kB/s | 45 kB 00:00 2024-09-20T19:33:09.518 INFO:teuthology.orchestra.run.smithi160.stdout:(7/32): python3-ceph-common-19.1.1-369.ged8ee0f 2.0 MB/s | 136 kB 00:00 2024-09-20T19:33:09.744 INFO:teuthology.orchestra.run.smithi160.stdout:(8/32): ceph-common-19.1.1-369.ged8ee0f2.el9.x8 17 MB/s | 21 MB 00:01 2024-09-20T19:33:09.769 INFO:teuthology.orchestra.run.smithi160.stdout:(9/32): python3-cephfs-19.1.1-369.ged8ee0f2.el9 646 kB/s | 162 kB 00:00 2024-09-20T19:33:09.794 INFO:teuthology.orchestra.run.smithi160.stdout:(10/32): python3-rados-19.1.1-369.ged8ee0f2.el9 6.3 MB/s | 320 kB 00:00 2024-09-20T19:33:09.870 INFO:teuthology.orchestra.run.smithi160.stdout:(11/32): librgw2-19.1.1-369.ged8ee0f2.el9.x86_6 10 MB/s | 5.3 MB 00:00 2024-09-20T19:33:09.895 INFO:teuthology.orchestra.run.smithi160.stdout:(12/32): python3-rbd-19.1.1-369.ged8ee0f2.el9.x 2.3 MB/s | 301 kB 00:00 2024-09-20T19:33:09.920 INFO:teuthology.orchestra.run.smithi160.stdout:(13/32): python3-rgw-19.1.1-369.ged8ee0f2.el9.x 792 kB/s | 99 kB 00:00 2024-09-20T19:33:10.004 INFO:teuthology.orchestra.run.smithi160.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 798 kB/s | 107 kB 00:00 2024-09-20T19:33:10.029 INFO:teuthology.orchestra.run.smithi160.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 417 kB/s | 45 kB 00:00 2024-09-20T19:33:10.054 INFO:teuthology.orchestra.run.smithi160.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-20T19:33:10.096 INFO:teuthology.orchestra.run.smithi160.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-20T19:33:10.130 INFO:teuthology.orchestra.run.smithi160.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-20T19:33:10.155 INFO:teuthology.orchestra.run.smithi160.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-09-20T19:33:10.180 INFO:teuthology.orchestra.run.smithi160.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 716 kB/s | 36 kB 00:00 2024-09-20T19:33:10.206 INFO:teuthology.orchestra.run.smithi160.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-20T19:33:10.231 INFO:teuthology.orchestra.run.smithi160.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 628 kB/s | 47 kB 00:00 2024-09-20T19:33:10.290 INFO:teuthology.orchestra.run.smithi160.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-20T19:33:10.315 INFO:teuthology.orchestra.run.smithi160.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 378 kB/s | 26 kB 00:00 2024-09-20T19:33:10.340 INFO:teuthology.orchestra.run.smithi160.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 1.0 MB/s | 49 kB 00:00 2024-09-20T19:33:10.365 INFO:teuthology.orchestra.run.smithi160.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-20T19:33:10.399 INFO:teuthology.orchestra.run.smithi160.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-20T19:33:10.441 INFO:teuthology.orchestra.run.smithi160.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-09-20T19:33:10.491 INFO:teuthology.orchestra.run.smithi160.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-20T19:33:10.559 INFO:teuthology.orchestra.run.smithi160.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-20T19:33:10.667 INFO:teuthology.orchestra.run.smithi160.stdout:(31/32): librados2-19.1.1-369.ged8ee0f2.el9.x86 15 MB/s | 3.4 MB 00:00 2024-09-20T19:33:10.726 INFO:teuthology.orchestra.run.smithi160.stdout:(32/32): librbd1-19.1.1-369.ged8ee0f2.el9.x86_6 14 MB/s | 3.2 MB 00:00 2024-09-20T19:33:10.731 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:33:10.731 INFO:teuthology.orchestra.run.smithi160.stdout:Total 25 MB/s | 60 MB 00:02 2024-09-20T19:33:10.831 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:33:10.868 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:33:10.868 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:33:11.317 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:33:11.338 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:33:11.338 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-20T19:33:11.339 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:33:11.339 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:33:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 11 M 2024-09-20T19:33:11.339 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-09-20T19:33:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 3.4 M 2024-09-20T19:33:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 3.2 M 2024-09-20T19:33:11.339 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-20T19:33:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-20T19:33:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 5.4 M 2024-09-20T19:33:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 21 M 2024-09-20T19:33:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 25 k 2024-09-20T19:33:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-20T19:33:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-20T19:33:11.339 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-20T19:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 751 k 2024-09-20T19:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-20T19:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-20T19:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-20T19:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 506 k 2024-09-20T19:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-20T19:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 5.3 M 2024-09-20T19:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-20T19:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-20T19:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-20T19:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 45 k 2024-09-20T19:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 136 k 2024-09-20T19:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 162 k 2024-09-20T19:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-20T19:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 320 k 2024-09-20T19:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 301 k 2024-09-20T19:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 99 k 2024-09-20T19:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-20T19:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-20T19:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-20T19:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-20T19:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-09-20T19:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-20T19:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout:Install 30 Packages 2024-09-20T19:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 2 Packages 2024-09-20T19:33:11.342 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 60 M 2024-09-20T19:33:11.342 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:33:11.557 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:33:11.558 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:33:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:(1/32): ceph-base-19.1.1-369.ged8ee0f2.el9.x86_ 11 MB/s | 5.4 MB 00:00 2024-09-20T19:33:12.084 INFO:teuthology.orchestra.run.smithi114.stdout:(2/32): ceph-selinux-19.1.1-369.ged8ee0f2.el9.x 503 kB/s | 25 kB 00:00 2024-09-20T19:33:12.184 INFO:teuthology.orchestra.run.smithi114.stdout:(3/32): libcephfs2-19.1.1-369.ged8ee0f2.el9.x86 7.3 MB/s | 751 kB 00:00 2024-09-20T19:33:12.284 INFO:teuthology.orchestra.run.smithi114.stdout:(4/32): libradosstriper1-19.1.1-369.ged8ee0f2.e 5.0 MB/s | 506 kB 00:00 2024-09-20T19:33:12.401 INFO:teuthology.orchestra.run.smithi114.stdout:(5/32): ceph-radosgw-19.1.1-369.ged8ee0f2.el9.x 12 MB/s | 11 MB 00:00 2024-09-20T19:33:12.452 INFO:teuthology.orchestra.run.smithi114.stdout:(6/32): python3-ceph-argparse-19.1.1-369.ged8ee 904 kB/s | 45 kB 00:00 2024-09-20T19:33:12.585 INFO:teuthology.orchestra.run.smithi114.stdout:(7/32): librgw2-19.1.1-369.ged8ee0f2.el9.x86_64 18 MB/s | 5.3 MB 00:00 2024-09-20T19:33:12.611 INFO:teuthology.orchestra.run.smithi114.stdout:(8/32): python3-ceph-common-19.1.1-369.ged8ee0f 859 kB/s | 136 kB 00:00 2024-09-20T19:33:12.636 INFO:teuthology.orchestra.run.smithi114.stdout:(9/32): python3-cephfs-19.1.1-369.ged8ee0f2.el9 3.1 MB/s | 162 kB 00:00 2024-09-20T19:33:12.845 INFO:teuthology.orchestra.run.smithi114.stdout:(10/32): ceph-common-19.1.1-369.ged8ee0f2.el9.x 16 MB/s | 21 MB 00:01 2024-09-20T19:33:12.878 INFO:teuthology.orchestra.run.smithi114.stdout:(11/32): python3-rados-19.1.1-369.ged8ee0f2.el9 1.2 MB/s | 320 kB 00:00 2024-09-20T19:33:12.903 INFO:teuthology.orchestra.run.smithi114.stdout:(12/32): python3-rgw-19.1.1-369.ged8ee0f2.el9.x 1.7 MB/s | 99 kB 00:00 2024-09-20T19:33:12.937 INFO:teuthology.orchestra.run.smithi114.stdout:(13/32): python3-rbd-19.1.1-369.ged8ee0f2.el9.x 1.0 MB/s | 301 kB 00:00 2024-09-20T19:33:13.004 INFO:teuthology.orchestra.run.smithi114.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 851 kB/s | 107 kB 00:00 2024-09-20T19:33:13.038 INFO:teuthology.orchestra.run.smithi114.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 451 kB/s | 45 kB 00:00 2024-09-20T19:33:13.063 INFO:teuthology.orchestra.run.smithi114.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-20T19:33:13.105 INFO:teuthology.orchestra.run.smithi114.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-09-20T19:33:13.117 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:33:13.130 INFO:teuthology.orchestra.run.smithi114.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 630 kB/s | 42 kB 00:00 2024-09-20T19:33:13.164 INFO:teuthology.orchestra.run.smithi114.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-20T19:33:13.189 INFO:teuthology.orchestra.run.smithi114.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-20T19:33:13.201 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-20T19:33:13.214 INFO:teuthology.orchestra.run.smithi114.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 428 kB/s | 36 kB 00:00 2024-09-20T19:33:13.230 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-20T19:33:13.239 INFO:teuthology.orchestra.run.smithi114.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-09-20T19:33:13.264 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-20T19:33:13.286 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-20T19:33:13.298 INFO:teuthology.orchestra.run.smithi114.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-20T19:33:13.311 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-20T19:33:13.323 INFO:teuthology.orchestra.run.smithi114.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-20T19:33:13.349 INFO:teuthology.orchestra.run.smithi114.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 276 kB/s | 26 kB 00:00 2024-09-20T19:33:13.374 INFO:teuthology.orchestra.run.smithi114.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-20T19:33:13.408 INFO:teuthology.orchestra.run.smithi114.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-20T19:33:13.458 INFO:teuthology.orchestra.run.smithi114.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-20T19:33:13.473 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-20T19:33:13.500 INFO:teuthology.orchestra.run.smithi114.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-20T19:33:13.527 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-20T19:33:13.584 INFO:teuthology.orchestra.run.smithi114.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-20T19:33:13.684 INFO:teuthology.orchestra.run.smithi114.stdout:(31/32): librados2-19.1.1-369.ged8ee0f2.el9.x86 15 MB/s | 3.4 MB 00:00 2024-09-20T19:33:13.751 INFO:teuthology.orchestra.run.smithi114.stdout:(32/32): librbd1-19.1.1-369.ged8ee0f2.el9.x86_6 13 MB/s | 3.2 MB 00:00 2024-09-20T19:33:13.755 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:33:13.756 INFO:teuthology.orchestra.run.smithi114.stdout:Total 25 MB/s | 60 MB 00:02 2024-09-20T19:33:13.854 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:33:13.885 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-20T19:33:13.889 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:33:13.889 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:33:13.991 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librados2-2:19.1.1-369.ged8ee0f2.el9.x86_64 7/34 2024-09-20T19:33:14.083 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:19.1.1-369.ged8ee0f2.el9.x86_64 7/34 2024-09-20T19:33:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rados-2:19.1.1-369.ged8ee0f2.el9.x86_64 8/34 2024-09-20T19:33:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs2-2:19.1.1-369.ged8ee0f2.el9.x86_64 9/34 2024-09-20T19:33:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs2-2:19.1.1-369.ged8ee0f2.el9.x86_64 9/34 2024-09-20T19:33:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-20T19:33:14.328 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-argparse-2:19.1.1-369.ged8ee0f2.el9.x 11/34 2024-09-20T19:33:14.406 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cephfs-2:19.1.1-369.ged8ee0f2.el9.x86_64 12/34 2024-09-20T19:33:14.436 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libradosstriper1-2:19.1.1-369.ged8ee0f2.el9.x86_64 13/34 2024-09-20T19:33:14.500 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libradosstriper1-2:19.1.1-369.ged8ee0f2.el9.x86_64 13/34 2024-09-20T19:33:14.535 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-20T19:33:14.561 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:33:14.561 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:33:14.647 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-20T19:33:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-20T19:33:14.736 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-20T19:33:14.775 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-20T19:33:14.805 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-20T19:33:14.838 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-20T19:33:15.149 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-20T19:33:15.192 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librbd1-2:19.1.1-369.ged8ee0f2.el9.x86_64 22/34 2024-09-20T19:33:15.278 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:19.1.1-369.ged8ee0f2.el9.x86_64 22/34 2024-09-20T19:33:15.314 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 23/34 2024-09-20T19:33:15.358 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-20T19:33:15.842 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-20T19:33:15.869 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librgw2-2:19.1.1-369.ged8ee0f2.el9.x86_64 26/34 2024-09-20T19:33:15.924 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librgw2-2:19.1.1-369.ged8ee0f2.el9.x86_64 26/34 2024-09-20T19:33:16.002 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 27/34 2024-09-20T19:33:16.055 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:33:16.055 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86 28/34 2024-09-20T19:33:16.135 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-20T19:33:16.164 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-20T19:33:16.189 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-20T19:33:16.212 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-20T19:33:16.234 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-20T19:33:16.414 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-20T19:33:16.467 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-20T19:33:16.816 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-20T19:33:17.008 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:19.1.1-369.ged8ee0f2.el9.x86_64 7/34 2024-09-20T19:33:17.101 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:19.1.1-369.ged8ee0f2.el9.x86_64 7/34 2024-09-20T19:33:17.198 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:19.1.1-369.ged8ee0f2.el9.x86_64 8/34 2024-09-20T19:33:17.234 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:19.1.1-369.ged8ee0f2.el9.x86_64 9/34 2024-09-20T19:33:17.297 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:19.1.1-369.ged8ee0f2.el9.x86_64 9/34 2024-09-20T19:33:17.326 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-20T19:33:17.370 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:19.1.1-369.ged8ee0f2.el9.x 11/34 2024-09-20T19:33:17.441 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:19.1.1-369.ged8ee0f2.el9.x86_64 12/34 2024-09-20T19:33:17.470 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:19.1.1-369.ged8ee0f2.el9.x86_64 13/34 2024-09-20T19:33:17.534 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:19.1.1-369.ged8ee0f2.el9.x86_64 13/34 2024-09-20T19:33:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-20T19:33:17.679 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-20T19:33:17.720 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-20T19:33:17.760 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-20T19:33:17.801 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-20T19:33:17.830 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-20T19:33:17.876 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-20T19:33:18.192 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-20T19:33:18.234 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:19.1.1-369.ged8ee0f2.el9.x86_64 22/34 2024-09-20T19:33:18.327 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:19.1.1-369.ged8ee0f2.el9.x86_64 22/34 2024-09-20T19:33:18.355 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 23/34 2024-09-20T19:33:18.373 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86_64 29/34 2024-09-20T19:33:18.391 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-20T19:33:18.409 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86_64 29/34 2024-09-20T19:33:18.872 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-20T19:33:18.903 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:19.1.1-369.ged8ee0f2.el9.x86_64 26/34 2024-09-20T19:33:18.980 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:19.1.1-369.ged8ee0f2.el9.x86_64 26/34 2024-09-20T19:33:18.981 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86_64 29/34 2024-09-20T19:33:19.038 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-base-2:19.1.1-369.ged8ee0f2.el9.x86_64 30/34 2024-09-20T19:33:19.051 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 27/34 2024-09-20T19:33:19.091 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86 28/34 2024-09-20T19:33:19.127 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-base-2:19.1.1-369.ged8ee0f2.el9.x86_64 30/34 2024-09-20T19:33:19.127 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-20T19:33:19.127 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-20T19:33:19.127 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:19.180 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-selinux-2:19.1.1-369.ged8ee0f2.el9.x86_64 31/34 2024-09-20T19:33:21.390 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86_64 29/34 2024-09-20T19:33:21.429 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86_64 29/34 2024-09-20T19:33:22.000 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86_64 29/34 2024-09-20T19:33:22.041 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:19.1.1-369.ged8ee0f2.el9.x86_64 30/34 2024-09-20T19:33:22.155 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:19.1.1-369.ged8ee0f2.el9.x86_64 30/34 2024-09-20T19:33:22.155 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-20T19:33:22.155 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-20T19:33:22.155 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:22.184 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:19.1.1-369.ged8ee0f2.el9.x86_64 31/34 2024-09-20T19:33:29.158 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-selinux-2:19.1.1-369.ged8ee0f2.el9.x86_64 31/34 2024-09-20T19:33:29.159 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /sys 2024-09-20T19:33:29.159 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /proc 2024-09-20T19:33:29.159 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /mnt 2024-09-20T19:33:29.159 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /var/tmp 2024-09-20T19:33:29.159 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /home 2024-09-20T19:33:29.159 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /root 2024-09-20T19:33:29.159 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /tmp 2024-09-20T19:33:29.159 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:30.188 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-radosgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 32/34 2024-09-20T19:33:30.214 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 32/34 2024-09-20T19:33:30.214 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T19:33:30.214 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-20T19:33:30.214 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-20T19:33:30.214 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-20T19:33:30.214 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:30.216 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-20T19:33:30.291 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-20T19:33:30.291 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-20T19:33:31.501 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-20T19:33:31.501 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-base-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/34 2024-09-20T19:33:31.501 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86_64 2/34 2024-09-20T19:33:31.502 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-radosgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 3/34 2024-09-20T19:33:31.502 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-selinux-2:19.1.1-369.ged8ee0f2.el9.x86_64 4/34 2024-09-20T19:33:31.502 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs2-2:19.1.1-369.ged8ee0f2.el9.x86_64 5/34 2024-09-20T19:33:31.502 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libradosstriper1-2:19.1.1-369.ged8ee0f2.el9.x86_64 6/34 2024-09-20T19:33:31.502 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librgw2-2:19.1.1-369.ged8ee0f2.el9.x86_64 7/34 2024-09-20T19:33:31.502 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-argparse-2:19.1.1-369.ged8ee0f2.el9.x 8/34 2024-09-20T19:33:31.502 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86 9/34 2024-09-20T19:33:31.502 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cephfs-2:19.1.1-369.ged8ee0f2.el9.x86_64 10/34 2024-09-20T19:33:31.502 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rados-2:19.1.1-369.ged8ee0f2.el9.x86_64 11/34 2024-09-20T19:33:31.502 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 12/34 2024-09-20T19:33:31.502 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 13/34 2024-09-20T19:33:31.502 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-20T19:33:31.502 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-20T19:33:31.503 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-20T19:33:31.503 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-20T19:33:31.503 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-20T19:33:31.503 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-20T19:33:31.503 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-20T19:33:31.503 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-20T19:33:31.503 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-20T19:33:31.503 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-20T19:33:31.503 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-20T19:33:31.503 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-20T19:33:31.503 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-20T19:33:31.503 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-20T19:33:31.503 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-20T19:33:31.503 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-20T19:33:31.504 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-20T19:33:31.504 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:19.1.1-369.ged8ee0f2.el9.x86_64 31/34 2024-09-20T19:33:31.504 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-20T19:33:31.504 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:19.1.1-369.ged8ee0f2.el9.x86_64 33/34 2024-09-20T19:33:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:19.1.1-369.ged8ee0f2.el9.x86_64 31/34 2024-09-20T19:33:32.229 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-09-20T19:33:32.229 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-09-20T19:33:32.229 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-09-20T19:33:32.229 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-09-20T19:33:32.229 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-09-20T19:33:32.229 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-09-20T19:33:32.229 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-09-20T19:33:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:32.515 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-20T19:33:32.515 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:32.515 INFO:teuthology.orchestra.run.smithi160.stdout:Upgraded: 2024-09-20T19:33:32.515 INFO:teuthology.orchestra.run.smithi160.stdout: librados2-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:32.515 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:32.515 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:33:32.515 INFO:teuthology.orchestra.run.smithi160.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-20T19:33:32.515 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:32.515 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:32.515 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:32.515 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:32.515 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-20T19:33:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-20T19:33:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-20T19:33:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-20T19:33:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-20T19:33:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-20T19:33:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-20T19:33:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-20T19:33:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-20T19:33:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-20T19:33:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:32.517 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:32.517 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-20T19:33:32.517 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:32.517 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:32.517 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:32.517 INFO:teuthology.orchestra.run.smithi160.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-20T19:33:32.517 INFO:teuthology.orchestra.run.smithi160.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-20T19:33:32.517 INFO:teuthology.orchestra.run.smithi160.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-20T19:33:32.517 INFO:teuthology.orchestra.run.smithi160.stdout: re2-1:20211101-20.el9.x86_64 2024-09-20T19:33:32.517 INFO:teuthology.orchestra.run.smithi160.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-20T19:33:32.517 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:32.517 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:33:32.892 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-test 2024-09-20T19:33:33.199 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 32/34 2024-09-20T19:33:33.227 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 32/34 2024-09-20T19:33:33.227 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T19:33:33.227 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-20T19:33:33.227 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-20T19:33:33.227 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-20T19:33:33.227 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:33.229 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-20T19:33:33.318 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-20T19:33:33.318 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-20T19:33:33.475 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:28 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:33:33.611 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:33:33.612 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:33:33.612 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-20T19:33:33.612 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:33:33.612 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:33:33.612 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 49 M 2024-09-20T19:33:33.612 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-20T19:33:33.612 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 163 k 2024-09-20T19:33:33.612 INFO:teuthology.orchestra.run.smithi160.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-20T19:33:33.612 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-20T19:33:33.612 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:33.612 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:33:33.613 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:33:33.613 INFO:teuthology.orchestra.run.smithi160.stdout:Install 4 Packages 2024-09-20T19:33:33.613 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:33.613 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 49 M 2024-09-20T19:33:33.613 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 209 M 2024-09-20T19:33:33.613 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:33:33.945 INFO:teuthology.orchestra.run.smithi160.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-20T19:33:33.979 INFO:teuthology.orchestra.run.smithi160.stdout:(2/4): libcephsqlite-19.1.1-369.ged8ee0f2.el9.x 708 kB/s | 163 kB 00:00 2024-09-20T19:33:34.005 INFO:teuthology.orchestra.run.smithi160.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-20T19:33:34.620 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-20T19:33:34.620 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/34 2024-09-20T19:33:34.620 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86_64 2/34 2024-09-20T19:33:34.620 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 3/34 2024-09-20T19:33:34.620 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:19.1.1-369.ged8ee0f2.el9.x86_64 4/34 2024-09-20T19:33:34.620 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:19.1.1-369.ged8ee0f2.el9.x86_64 5/34 2024-09-20T19:33:34.620 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:19.1.1-369.ged8ee0f2.el9.x86_64 6/34 2024-09-20T19:33:34.620 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:19.1.1-369.ged8ee0f2.el9.x86_64 7/34 2024-09-20T19:33:34.620 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:19.1.1-369.ged8ee0f2.el9.x 8/34 2024-09-20T19:33:34.620 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86 9/34 2024-09-20T19:33:34.620 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:19.1.1-369.ged8ee0f2.el9.x86_64 10/34 2024-09-20T19:33:34.620 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:19.1.1-369.ged8ee0f2.el9.x86_64 11/34 2024-09-20T19:33:34.621 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 12/34 2024-09-20T19:33:34.621 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 13/34 2024-09-20T19:33:34.621 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-20T19:33:34.621 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-20T19:33:34.621 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-20T19:33:34.621 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-20T19:33:34.621 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-20T19:33:34.621 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-20T19:33:34.621 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-20T19:33:34.621 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-20T19:33:34.621 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-20T19:33:34.621 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-20T19:33:34.621 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-20T19:33:34.621 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-20T19:33:34.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-20T19:33:34.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-20T19:33:34.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-20T19:33:34.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-20T19:33:34.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-20T19:33:34.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:19.1.1-369.ged8ee0f2.el9.x86_64 31/34 2024-09-20T19:33:34.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-20T19:33:34.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:19.1.1-369.ged8ee0f2.el9.x86_64 33/34 2024-09-20T19:33:35.279 INFO:teuthology.orchestra.run.smithi160.stdout:(4/4): ceph-test-19.1.1-369.ged8ee0f2.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-20T19:33:35.281 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:33:35.282 INFO:teuthology.orchestra.run.smithi160.stdout:Total 29 MB/s | 49 MB 00:01 2024-09-20T19:33:35.310 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:33:35.330 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:33:35.331 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:33:35.720 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:33:35.721 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:33:35.846 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-20T19:33:35.846 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:35.846 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-09-20T19:33:35.846 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:35.847 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:35.847 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:33:35.847 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-20T19:33:35.847 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:35.847 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:35.847 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:35.847 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:35.847 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-20T19:33:35.847 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-20T19:33:35.847 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-20T19:33:35.847 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:35.847 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-20T19:33:35.847 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-20T19:33:35.847 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-20T19:33:35.848 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:35.848 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-20T19:33:35.848 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:35.848 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-20T19:33:35.848 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-20T19:33:35.848 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-20T19:33:35.848 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:35.848 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:35.848 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:35.848 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-20T19:33:35.848 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:35.848 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:35.848 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:35.848 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-20T19:33:35.849 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-20T19:33:35.849 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-20T19:33:35.849 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-20.el9.x86_64 2024-09-20T19:33:35.849 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-20T19:33:35.849 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:35.849 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:33:36.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-test 2024-09-20T19:33:36.252 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:33:36.331 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-20T19:33:36.375 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-20T19:33:36.406 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephsqlite-2:19.1.1-369.ged8ee0f2.el9.x86_64 3/4 2024-09-20T19:33:36.694 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:27 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:33:36.836 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:33:36.836 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:33:36.836 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-20T19:33:36.836 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:33:36.837 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:33:36.837 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 49 M 2024-09-20T19:33:36.837 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-20T19:33:36.837 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 163 k 2024-09-20T19:33:36.837 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-20T19:33:36.837 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-20T19:33:36.837 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:36.837 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:33:36.837 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:33:36.837 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-09-20T19:33:36.837 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:36.838 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 49 M 2024-09-20T19:33:36.838 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 209 M 2024-09-20T19:33:36.838 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:33:37.017 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-09-20T19:33:37.051 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-09-20T19:33:37.101 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): libcephsqlite-19.1.1-369.ged8ee0f2.el9.x 682 kB/s | 163 kB 00:00 2024-09-20T19:33:38.376 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): ceph-test-19.1.1-369.ged8ee0f2.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-20T19:33:38.378 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:33:38.379 INFO:teuthology.orchestra.run.smithi114.stdout:Total 32 MB/s | 49 MB 00:01 2024-09-20T19:33:38.407 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:33:38.427 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:33:38.427 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:33:38.826 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:33:38.827 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:33:39.361 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:33:39.445 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-20T19:33:39.484 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-20T19:33:39.519 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:19.1.1-369.ged8ee0f2.el9.x86_64 3/4 2024-09-20T19:33:41.018 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephsqlite-2:19.1.1-369.ged8ee0f2.el9.x86_64 3/4 2024-09-20T19:33:41.104 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-test-2:19.1.1-369.ged8ee0f2.el9.x86_64 4/4 2024-09-20T19:33:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-test-2:19.1.1-369.ged8ee0f2.el9.x86_64 4/4 2024-09-20T19:33:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-test-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/4 2024-09-20T19:33:43.467 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephsqlite-2:19.1.1-369.ged8ee0f2.el9.x86_64 2/4 2024-09-20T19:33:43.468 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-20T19:33:43.770 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-20T19:33:43.770 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:43.770 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:33:43.770 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:43.770 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:43.770 INFO:teuthology.orchestra.run.smithi160.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-20T19:33:43.770 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-20T19:33:43.770 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:43.770 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:33:44.073 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph 2024-09-20T19:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:19.1.1-369.ged8ee0f2.el9.x86_64 3/4 2024-09-20T19:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:19.1.1-369.ged8ee0f2.el9.x86_64 4/4 2024-09-20T19:33:44.681 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:39 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:33:44.825 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:33:44.827 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:33:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-20T19:33:44.827 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:33:44.827 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:33:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: ceph x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 6.5 k 2024-09-20T19:33:44.827 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-20T19:33:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 2.4 M 2024-09-20T19:33:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 1.5 M 2024-09-20T19:33:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 249 k 2024-09-20T19:33:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 4.7 M 2024-09-20T19:33:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 17 M 2024-09-20T19:33:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-20T19:33:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-20T19:33:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-20T19:33:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-20T19:33:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-20T19:33:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-20T19:33:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-20T19:33:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-20T19:33:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-20T19:33:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-20T19:33:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-20T19:33:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-20T19:33:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-20T19:33:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-20T19:33:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-20T19:33:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-20T19:33:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-20T19:33:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-20T19:33:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-20T19:33:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-20T19:33:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-20T19:33:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-20T19:33:44.830 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-20T19:33:44.830 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-20T19:33:44.830 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-20T19:33:44.830 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-20T19:33:44.830 INFO:teuthology.orchestra.run.smithi160.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-20T19:33:44.830 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-20T19:33:44.830 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-20T19:33:44.830 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-20T19:33:44.830 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-20T19:33:44.830 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:44.830 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:33:44.830 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:33:44.830 INFO:teuthology.orchestra.run.smithi160.stdout:Install 37 Packages 2024-09-20T19:33:44.830 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:44.831 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 30 M 2024-09-20T19:33:44.831 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 108 M 2024-09-20T19:33:44.831 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:33:45.700 INFO:teuthology.orchestra.run.smithi160.stdout:(1/37): ceph-19.1.1-369.ged8ee0f2.el9.x86_64.rp 54 kB/s | 6.5 kB 00:00 2024-09-20T19:33:45.875 INFO:teuthology.orchestra.run.smithi160.stdout:(2/37): ceph-mgr-19.1.1-369.ged8ee0f2.el9.x86_6 5.1 MB/s | 1.5 MB 00:00 2024-09-20T19:33:45.942 INFO:teuthology.orchestra.run.smithi160.stdout:(3/37): ceph-mds-19.1.1-369.ged8ee0f2.el9.x86_6 6.6 MB/s | 2.4 MB 00:00 2024-09-20T19:33:46.034 INFO:teuthology.orchestra.run.smithi160.stdout:(4/37): ceph-mon-19.1.1-369.ged8ee0f2.el9.x86_6 14 MB/s | 4.7 MB 00:00 2024-09-20T19:33:46.059 INFO:teuthology.orchestra.run.smithi160.stdout:(5/37): ceph-mgr-modules-core-19.1.1-369.ged8ee 2.1 MB/s | 249 kB 00:00 2024-09-20T19:33:46.426 INFO:teuthology.orchestra.run.smithi160.stdout:(6/37): ceph-osd-19.1.1-369.ged8ee0f2.el9.x86_6 31 MB/s | 17 MB 00:00 2024-09-20T19:33:46.457 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-test-2:19.1.1-369.ged8ee0f2.el9.x86_64 4/4 2024-09-20T19:33:46.457 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/4 2024-09-20T19:33:46.458 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:19.1.1-369.ged8ee0f2.el9.x86_64 2/4 2024-09-20T19:33:46.458 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-20T19:33:46.677 INFO:teuthology.orchestra.run.smithi160.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 394 kB/s | 253 kB 00:00 2024-09-20T19:33:46.782 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-20T19:33:46.782 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:46.782 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:33:46.782 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:46.783 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:46.783 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-20T19:33:46.783 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-20T19:33:46.783 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:46.783 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:33:46.785 INFO:teuthology.orchestra.run.smithi160.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.2 MB/s | 135 kB 00:00 2024-09-20T19:33:46.894 INFO:teuthology.orchestra.run.smithi160.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 228 kB/s | 106 kB 00:00 2024-09-20T19:33:46.919 INFO:teuthology.orchestra.run.smithi160.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 935 kB/s | 125 kB 00:00 2024-09-20T19:33:46.969 INFO:teuthology.orchestra.run.smithi160.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-09-20T19:33:47.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph 2024-09-20T19:33:47.095 INFO:teuthology.orchestra.run.smithi160.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-09-20T19:33:47.128 INFO:teuthology.orchestra.run.smithi160.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 902 kB/s | 188 kB 00:00 2024-09-20T19:33:47.162 INFO:teuthology.orchestra.run.smithi160.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 896 kB/s | 172 kB 00:00 2024-09-20T19:33:47.187 INFO:teuthology.orchestra.run.smithi160.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.3 MB/s | 77 kB 00:00 2024-09-20T19:33:47.212 INFO:teuthology.orchestra.run.smithi160.stdout:(16/37): python3-markupsafe-1.1.1-12.el9.x86_64 296 kB/s | 35 kB 00:00 2024-09-20T19:33:47.238 INFO:teuthology.orchestra.run.smithi160.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 677 kB/s | 51 kB 00:00 2024-09-20T19:33:47.271 INFO:teuthology.orchestra.run.smithi160.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 497 kB/s | 42 kB 00:00 2024-09-20T19:33:47.322 INFO:teuthology.orchestra.run.smithi160.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-09-20T19:33:47.347 INFO:teuthology.orchestra.run.smithi160.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 166 kB/s | 22 kB 00:00 2024-09-20T19:33:47.372 INFO:teuthology.orchestra.run.smithi160.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-20T19:33:47.406 INFO:teuthology.orchestra.run.smithi160.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-20T19:33:47.422 INFO:teuthology.orchestra.run.smithi160.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 247 kB/s | 11 kB 00:00 2024-09-20T19:33:47.448 INFO:teuthology.orchestra.run.smithi160.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 437 kB/s | 18 kB 00:00 2024-09-20T19:33:47.465 INFO:teuthology.orchestra.run.smithi160.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 549 kB/s | 23 kB 00:00 2024-09-20T19:33:47.482 INFO:teuthology.orchestra.run.smithi160.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 581 kB/s | 19 kB 00:00 2024-09-20T19:33:47.499 INFO:teuthology.orchestra.run.smithi160.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 585 kB/s | 20 kB 00:00 2024-09-20T19:33:47.524 INFO:teuthology.orchestra.run.smithi160.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-09-20T19:33:47.549 INFO:teuthology.orchestra.run.smithi160.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-20T19:33:47.575 INFO:teuthology.orchestra.run.smithi160.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 661 kB/s | 16 kB 00:00 2024-09-20T19:33:47.625 INFO:teuthology.orchestra.run.smithi160.stdout:(31/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-20T19:33:47.650 INFO:teuthology.orchestra.run.smithi160.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-20T19:33:47.676 INFO:teuthology.orchestra.run.smithi160.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-20T19:33:47.692 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:38 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:33:47.700 INFO:teuthology.orchestra.run.smithi160.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-09-20T19:33:47.726 INFO:teuthology.orchestra.run.smithi160.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 802 kB/s | 20 kB 00:00 2024-09-20T19:33:47.751 INFO:teuthology.orchestra.run.smithi160.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-20T19:33:47.776 INFO:teuthology.orchestra.run.smithi160.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-20T19:33:47.780 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:33:47.780 INFO:teuthology.orchestra.run.smithi160.stdout:Total 10 MB/s | 30 MB 00:02 2024-09-20T19:33:47.838 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:33:47.840 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:33:47.841 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-20T19:33:47.841 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:33:47.841 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:33:47.841 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 6.5 k 2024-09-20T19:33:47.841 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-20T19:33:47.841 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 2.4 M 2024-09-20T19:33:47.841 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 1.5 M 2024-09-20T19:33:47.841 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 249 k 2024-09-20T19:33:47.841 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 4.7 M 2024-09-20T19:33:47.841 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 17 M 2024-09-20T19:33:47.841 INFO:teuthology.orchestra.run.smithi114.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-20T19:33:47.841 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-20T19:33:47.842 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-20T19:33:47.842 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-20T19:33:47.842 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-20T19:33:47.842 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-20T19:33:47.842 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-20T19:33:47.842 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-20T19:33:47.842 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-20T19:33:47.842 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-20T19:33:47.842 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-20T19:33:47.842 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-20T19:33:47.842 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-20T19:33:47.842 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-20T19:33:47.842 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-20T19:33:47.843 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-20T19:33:47.843 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-20T19:33:47.843 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-20T19:33:47.843 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-20T19:33:47.843 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-20T19:33:47.843 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-20T19:33:47.843 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-20T19:33:47.843 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-20T19:33:47.843 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-20T19:33:47.843 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-20T19:33:47.843 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-20T19:33:47.843 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-20T19:33:47.843 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-20T19:33:47.843 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-20T19:33:47.844 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-20T19:33:47.844 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-20T19:33:47.844 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:47.844 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:33:47.844 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:33:47.844 INFO:teuthology.orchestra.run.smithi114.stdout:Install 37 Packages 2024-09-20T19:33:47.844 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:47.845 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 30 M 2024-09-20T19:33:47.845 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 108 M 2024-09-20T19:33:47.845 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:33:47.932 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:33:47.972 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:33:47.973 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:33:48.447 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:33:48.447 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:33:48.711 INFO:teuthology.orchestra.run.smithi114.stdout:(1/37): ceph-19.1.1-369.ged8ee0f2.el9.x86_64.rp 46 kB/s | 6.5 kB 00:00 2024-09-20T19:33:48.878 INFO:teuthology.orchestra.run.smithi114.stdout:(2/37): ceph-mgr-19.1.1-369.ged8ee0f2.el9.x86_6 4.9 MB/s | 1.5 MB 00:00 2024-09-20T19:33:48.945 INFO:teuthology.orchestra.run.smithi114.stdout:(3/37): ceph-mds-19.1.1-369.ged8ee0f2.el9.x86_6 6.3 MB/s | 2.4 MB 00:00 2024-09-20T19:33:49.020 INFO:teuthology.orchestra.run.smithi114.stdout:(4/37): ceph-mgr-modules-core-19.1.1-369.ged8ee 3.3 MB/s | 249 kB 00:00 2024-09-20T19:33:49.320 INFO:teuthology.orchestra.run.smithi114.stdout:(5/37): ceph-mon-19.1.1-369.ged8ee0f2.el9.x86_6 7.7 MB/s | 4.7 MB 00:00 2024-09-20T19:33:49.479 INFO:teuthology.orchestra.run.smithi114.stdout:(6/37): ceph-osd-19.1.1-369.ged8ee0f2.el9.x86_6 28 MB/s | 17 MB 00:00 2024-09-20T19:33:49.654 INFO:teuthology.orchestra.run.smithi114.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 399 kB/s | 253 kB 00:00 2024-09-20T19:33:49.834 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:33:49.855 INFO:teuthology.orchestra.run.smithi114.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 676 kB/s | 135 kB 00:00 2024-09-20T19:33:49.896 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-20T19:33:49.937 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-20T19:33:49.938 INFO:teuthology.orchestra.run.smithi114.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 232 kB/s | 106 kB 00:00 2024-09-20T19:33:49.963 INFO:teuthology.orchestra.run.smithi114.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-09-20T19:33:49.973 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-20T19:33:50.046 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-20T19:33:50.080 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-20T19:33:50.113 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-20T19:33:50.147 INFO:teuthology.orchestra.run.smithi114.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-09-20T19:33:50.172 INFO:teuthology.orchestra.run.smithi114.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 904 kB/s | 188 kB 00:00 2024-09-20T19:33:50.214 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-20T19:33:50.222 INFO:teuthology.orchestra.run.smithi114.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 694 kB/s | 35 kB 00:00 2024-09-20T19:33:50.273 INFO:teuthology.orchestra.run.smithi114.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-20T19:33:50.293 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-20T19:33:50.323 INFO:teuthology.orchestra.run.smithi114.stdout:(15/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-20T19:33:50.348 INFO:teuthology.orchestra.run.smithi114.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 856 kB/s | 172 kB 00:00 2024-09-20T19:33:50.349 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-20T19:33:50.384 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-20T19:33:50.390 INFO:teuthology.orchestra.run.smithi114.stdout:(17/37): python3-cryptography-36.0.1-4.el9.x86_ 1.1 MB/s | 1.2 MB 00:01 2024-09-20T19:33:50.414 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-20T19:33:50.416 INFO:teuthology.orchestra.run.smithi114.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 451 kB/s | 42 kB 00:00 2024-09-20T19:33:50.449 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-20T19:33:50.458 INFO:teuthology.orchestra.run.smithi114.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-20T19:33:50.482 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-20T19:33:50.483 INFO:teuthology.orchestra.run.smithi114.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 166 kB/s | 22 kB 00:00 2024-09-20T19:33:50.516 INFO:teuthology.orchestra.run.smithi114.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-09-20T19:33:50.523 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-20T19:33:50.542 INFO:teuthology.orchestra.run.smithi114.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 603 kB/s | 11 kB 00:00 2024-09-20T19:33:50.566 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-20T19:33:50.567 INFO:teuthology.orchestra.run.smithi114.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-09-20T19:33:50.584 INFO:teuthology.orchestra.run.smithi114.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 442 kB/s | 18 kB 00:00 2024-09-20T19:33:50.600 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-20T19:33:50.601 INFO:teuthology.orchestra.run.smithi114.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 691 kB/s | 23 kB 00:00 2024-09-20T19:33:50.618 INFO:teuthology.orchestra.run.smithi114.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 579 kB/s | 19 kB 00:00 2024-09-20T19:33:50.635 INFO:teuthology.orchestra.run.smithi114.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 590 kB/s | 20 kB 00:00 2024-09-20T19:33:50.649 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-20T19:33:50.652 INFO:teuthology.orchestra.run.smithi114.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-09-20T19:33:50.678 INFO:teuthology.orchestra.run.smithi114.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-20T19:33:50.681 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-20T19:33:50.702 INFO:teuthology.orchestra.run.smithi114.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 662 kB/s | 16 kB 00:00 2024-09-20T19:33:50.736 INFO:teuthology.orchestra.run.smithi114.stdout:(31/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-20T19:33:50.761 INFO:teuthology.orchestra.run.smithi114.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-20T19:33:50.762 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-20T19:33:50.786 INFO:teuthology.orchestra.run.smithi114.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-09-20T19:33:50.812 INFO:teuthology.orchestra.run.smithi114.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-20T19:33:50.837 INFO:teuthology.orchestra.run.smithi114.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 805 kB/s | 20 kB 00:00 2024-09-20T19:33:50.862 INFO:teuthology.orchestra.run.smithi114.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-20T19:33:50.887 INFO:teuthology.orchestra.run.smithi114.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-20T19:33:50.892 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:33:50.892 INFO:teuthology.orchestra.run.smithi114.stdout:Total 9.9 MB/s | 30 MB 00:03 2024-09-20T19:33:50.909 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-20T19:33:51.006 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-20T19:33:51.047 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:33:51.058 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-20T19:33:51.088 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:33:51.088 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:33:51.097 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-20T19:33:51.154 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-20T19:33:51.210 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-20T19:33:51.407 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-20T19:33:51.451 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-20T19:33:51.529 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-20T19:33:51.592 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:33:51.592 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:33:51.617 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-20T19:33:51.659 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-20T19:33:51.786 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-20T19:33:51.957 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-369.ged8ee0f2.el9.n 32/37 2024-09-20T19:33:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-2:19.1.1-369.ged8ee0f2.el9.x86_64 33/37 2024-09-20T19:33:52.013 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-2:19.1.1-369.ged8ee0f2.el9.x86_64 33/37 2024-09-20T19:33:52.013 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T19:33:52.013 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-20T19:33:52.013 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-20T19:33:52.013 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-20T19:33:52.013 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:52.949 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:33:53.007 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-20T19:33:53.051 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-20T19:33:53.080 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-20T19:33:53.159 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-20T19:33:53.220 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-20T19:33:53.253 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-20T19:33:53.344 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-20T19:33:53.415 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-20T19:33:53.453 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-osd-2:19.1.1-369.ged8ee0f2.el9.x86_64 34/37 2024-09-20T19:33:53.464 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-20T19:33:53.481 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-osd-2:19.1.1-369.ged8ee0f2.el9.x86_64 34/37 2024-09-20T19:33:53.481 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T19:33:53.481 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-20T19:33:53.481 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-20T19:33:53.481 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-20T19:33:53.482 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:53.499 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-20T19:33:53.529 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-20T19:33:53.564 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-20T19:33:53.597 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-20T19:33:53.638 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-20T19:33:53.673 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-20T19:33:53.715 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-20T19:33:53.764 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-20T19:33:53.795 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-20T19:33:53.869 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-20T19:33:53.929 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mon-2:19.1.1-369.ged8ee0f2.el9.x86_64 35/37 2024-09-20T19:33:53.953 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mon-2:19.1.1-369.ged8ee0f2.el9.x86_64 35/37 2024-09-20T19:33:53.953 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T19:33:53.953 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-20T19:33:53.953 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-20T19:33:53.953 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-20T19:33:53.953 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:54.015 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-20T19:33:54.095 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-20T19:33:54.148 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-20T19:33:54.180 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mds-2:19.1.1-369.ged8ee0f2.el9.x86_64 36/37 2024-09-20T19:33:54.187 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-20T19:33:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mds-2:19.1.1-369.ged8ee0f2.el9.x86_64 36/37 2024-09-20T19:33:54.204 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T19:33:54.204 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-20T19:33:54.204 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-20T19:33:54.204 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-20T19:33:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:54.245 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-20T19:33:54.268 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-2:19.1.1-369.ged8ee0f2.el9.x86_64 37/37 2024-09-20T19:33:54.301 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-20T19:33:54.486 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-20T19:33:54.542 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-20T19:33:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-20T19:33:54.716 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-20T19:33:54.757 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-20T19:33:54.883 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-20T19:33:55.055 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-369.ged8ee0f2.el9.n 32/37 2024-09-20T19:33:55.085 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:19.1.1-369.ged8ee0f2.el9.x86_64 33/37 2024-09-20T19:33:55.111 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:19.1.1-369.ged8ee0f2.el9.x86_64 33/37 2024-09-20T19:33:55.111 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T19:33:55.111 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-20T19:33:55.111 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-20T19:33:55.111 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-20T19:33:55.111 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:56.083 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-2:19.1.1-369.ged8ee0f2.el9.x86_64 37/37 2024-09-20T19:33:56.083 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/37 2024-09-20T19:33:56.084 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mds-2:19.1.1-369.ged8ee0f2.el9.x86_64 2/37 2024-09-20T19:33:56.084 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-2:19.1.1-369.ged8ee0f2.el9.x86_64 3/37 2024-09-20T19:33:56.084 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mon-2:19.1.1-369.ged8ee0f2.el9.x86_64 4/37 2024-09-20T19:33:56.084 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-osd-2:19.1.1-369.ged8ee0f2.el9.x86_64 5/37 2024-09-20T19:33:56.084 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-369.ged8ee0f2.el9.n 6/37 2024-09-20T19:33:56.084 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-20T19:33:56.084 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-20T19:33:56.084 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-20T19:33:56.084 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-20T19:33:56.084 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-20T19:33:56.084 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-20T19:33:56.084 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-20T19:33:56.084 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-20T19:33:56.085 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-20T19:33:56.085 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-20T19:33:56.085 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-20T19:33:56.085 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-20T19:33:56.085 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-20T19:33:56.085 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-20T19:33:56.085 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-20T19:33:56.085 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-20T19:33:56.085 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-20T19:33:56.085 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-20T19:33:56.085 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-20T19:33:56.086 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-20T19:33:56.087 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-20T19:33:56.087 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-20T19:33:56.087 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-20T19:33:56.087 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-20T19:33:56.087 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-20T19:33:56.087 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-20T19:33:56.087 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-20T19:33:56.087 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-20T19:33:56.087 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-20T19:33:56.087 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-20T19:33:56.561 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:19.1.1-369.ged8ee0f2.el9.x86_64 34/37 2024-09-20T19:33:56.587 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:19.1.1-369.ged8ee0f2.el9.x86_64 34/37 2024-09-20T19:33:56.587 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T19:33:56.588 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-20T19:33:56.588 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-20T19:33:56.588 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-20T19:33:56.588 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:57.036 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:19.1.1-369.ged8ee0f2.el9.x86_64 35/37 2024-09-20T19:33:57.057 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-20T19:33:57.057 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:57.057 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:33:57.057 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:57.057 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:57.058 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:57.058 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:33:57.058 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:57.058 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:33:57.058 INFO:teuthology.orchestra.run.smithi160.stdout: lua-5.4.4-4.el9.x86_64 2024-09-20T19:33:57.058 INFO:teuthology.orchestra.run.smithi160.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-20T19:33:57.058 INFO:teuthology.orchestra.run.smithi160.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-20T19:33:57.058 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-20T19:33:57.058 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-20T19:33:57.058 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-20T19:33:57.058 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-20T19:33:57.058 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-20T19:33:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-20T19:33:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-20T19:33:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-20T19:33:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-20T19:33:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-20T19:33:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-20T19:33:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-20T19:33:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-20T19:33:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-20T19:33:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-20T19:33:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-20T19:33:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ply-3.11-14.el9.noarch 2024-09-20T19:33:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-20T19:33:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-20T19:33:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-20T19:33:57.060 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-20T19:33:57.060 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-20T19:33:57.060 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-20T19:33:57.060 INFO:teuthology.orchestra.run.smithi160.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-20T19:33:57.060 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-20T19:33:57.060 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-20T19:33:57.060 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-20T19:33:57.060 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-20T19:33:57.060 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:57.060 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:33:57.061 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:19.1.1-369.ged8ee0f2.el9.x86_64 35/37 2024-09-20T19:33:57.061 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T19:33:57.061 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-20T19:33:57.061 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-20T19:33:57.061 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-20T19:33:57.061 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:57.296 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:19.1.1-369.ged8ee0f2.el9.x86_64 36/37 2024-09-20T19:33:57.319 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:19.1.1-369.ged8ee0f2.el9.x86_64 36/37 2024-09-20T19:33:57.319 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T19:33:57.319 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-20T19:33:57.320 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-20T19:33:57.320 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-20T19:33:57.320 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:33:57.384 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:19.1.1-369.ged8ee0f2.el9.x86_64 37/37 2024-09-20T19:33:57.445 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-base 2024-09-20T19:33:58.046 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:53 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:33:58.136 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-base-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:33:58.179 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:33:58.180 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-20T19:33:58.180 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:33:58.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install cephadm 2024-09-20T19:33:58.835 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:53 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:33:58.967 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:33:58.968 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:33:58.968 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-20T19:33:58.968 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:33:58.968 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:33:58.968 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 765 k 2024-09-20T19:33:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:58.969 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:33:58.969 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:33:58.969 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-20T19:33:58.969 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:33:58.969 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 765 k 2024-09-20T19:33:58.969 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 771 k 2024-09-20T19:33:58.969 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:33:59.114 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-2:19.1.1-369.ged8ee0f2.el9.x86_64 37/37 2024-09-20T19:33:59.115 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/37 2024-09-20T19:33:59.115 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:19.1.1-369.ged8ee0f2.el9.x86_64 2/37 2024-09-20T19:33:59.115 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:19.1.1-369.ged8ee0f2.el9.x86_64 3/37 2024-09-20T19:33:59.115 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:19.1.1-369.ged8ee0f2.el9.x86_64 4/37 2024-09-20T19:33:59.115 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:19.1.1-369.ged8ee0f2.el9.x86_64 5/37 2024-09-20T19:33:59.115 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-369.ged8ee0f2.el9.n 6/37 2024-09-20T19:33:59.115 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-20T19:33:59.115 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-20T19:33:59.115 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-20T19:33:59.115 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-20T19:33:59.116 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-20T19:33:59.116 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-20T19:33:59.116 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-20T19:33:59.116 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-20T19:33:59.116 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-20T19:33:59.116 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-20T19:33:59.116 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-20T19:33:59.116 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-20T19:33:59.116 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-20T19:33:59.116 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-20T19:33:59.116 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-20T19:33:59.116 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-20T19:33:59.117 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-20T19:33:59.117 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-20T19:33:59.117 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-20T19:33:59.117 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-20T19:33:59.117 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-20T19:33:59.117 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-20T19:33:59.117 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-20T19:33:59.117 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-20T19:33:59.117 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-20T19:33:59.117 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-20T19:33:59.117 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-20T19:33:59.117 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-20T19:33:59.117 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-20T19:33:59.118 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-20T19:33:59.239 INFO:teuthology.orchestra.run.smithi160.stdout:cephadm-19.1.1-369.ged8ee0f2.el9.noarch.rpm 2.8 MB/s | 765 kB 00:00 2024-09-20T19:33:59.240 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:33:59.240 INFO:teuthology.orchestra.run.smithi160.stdout:Total 2.7 MB/s | 765 kB 00:00 2024-09-20T19:33:59.240 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:33:59.245 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:33:59.245 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:33:59.252 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:33:59.252 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:33:59.327 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:33:59.738 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 1/1 2024-09-20T19:33:59.827 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 1/1 2024-09-20T19:34:00.129 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-20T19:34:00.129 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:00.129 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:34:00.130 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:00.130 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:00.130 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:00.130 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:34:00.130 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:00.130 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:00.130 INFO:teuthology.orchestra.run.smithi114.stdout: lua-5.4.4-4.el9.x86_64 2024-09-20T19:34:00.130 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-20T19:34:00.130 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-20T19:34:00.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-20T19:34:00.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-20T19:34:00.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-20T19:34:00.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-20T19:34:00.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-20T19:34:00.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-20T19:34:00.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-20T19:34:00.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-20T19:34:00.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-20T19:34:00.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-20T19:34:00.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-20T19:34:00.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-20T19:34:00.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-20T19:34:00.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-20T19:34:00.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-20T19:34:00.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-20T19:34:00.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-09-20T19:34:00.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-20T19:34:00.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-20T19:34:00.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-20T19:34:00.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-20T19:34:00.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-20T19:34:00.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-20T19:34:00.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-20T19:34:00.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-20T19:34:00.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-20T19:34:00.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-20T19:34:00.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-20T19:34:00.132 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:00.132 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:00.236 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 1/1 2024-09-20T19:34:00.516 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 1/1 2024-09-20T19:34:00.516 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:00.516 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:34:00.517 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:34:00.517 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:00.517 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:00.555 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-base 2024-09-20T19:34:00.751 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-immutable-object-cache 2024-09-20T19:34:01.137 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:52 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:01.224 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-base-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:01.266 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:01.267 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-20T19:34:01.267 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:01.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install cephadm 2024-09-20T19:34:01.351 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:56 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:01.485 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:01.485 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:01.485 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-09-20T19:34:01.486 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:01.486 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:34:01.486 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 145 k 2024-09-20T19:34:01.486 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:01.486 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:34:01.486 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:01.486 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-20T19:34:01.486 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:01.487 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 145 k 2024-09-20T19:34:01.487 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 436 k 2024-09-20T19:34:01.487 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:34:01.703 INFO:teuthology.orchestra.run.smithi160.stdout:ceph-immutable-object-cache-19.1.1-369.ged8ee0f 671 kB/s | 145 kB 00:00 2024-09-20T19:34:01.703 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:01.704 INFO:teuthology.orchestra.run.smithi160.stdout:Total 667 kB/s | 145 kB 00:00 2024-09-20T19:34:01.704 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:34:01.711 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:34:01.712 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:34:01.769 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:34:01.769 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:34:01.906 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:52 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:01.911 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:34:01.970 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-369.ged8ee0f2.e 1/1 2024-09-20T19:34:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-369.ged8ee0f2.e 1/1 2024-09-20T19:34:01.996 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T19:34:01.996 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-20T19:34:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:02.036 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:02.036 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:02.036 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-20T19:34:02.036 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:02.036 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:34:02.037 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 765 k 2024-09-20T19:34:02.037 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:02.037 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:34:02.037 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:02.037 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-20T19:34:02.037 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:02.037 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 765 k 2024-09-20T19:34:02.037 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 771 k 2024-09-20T19:34:02.037 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:34:02.327 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm-19.1.1-369.ged8ee0f2.el9.noarch.rpm 2.6 MB/s | 765 kB 00:00 2024-09-20T19:34:02.328 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:02.328 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.6 MB/s | 765 kB 00:00 2024-09-20T19:34:02.328 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:34:02.334 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:34:02.334 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:34:02.340 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:34:02.340 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:34:02.409 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:34:02.834 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 1/1 2024-09-20T19:34:02.900 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-369.ged8ee0f2.e 1/1 2024-09-20T19:34:02.900 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:02.900 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:34:02.900 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:02.900 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:02.900 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:02.923 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 1/1 2024-09-20T19:34:03.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr 2024-09-20T19:34:03.291 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 1/1 2024-09-20T19:34:03.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 1/1 2024-09-20T19:34:03.573 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:03.574 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:34:03.574 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:34:03.574 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:03.574 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:03.743 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:58 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:03.772 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-immutable-object-cache 2024-09-20T19:34:03.832 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-mgr-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:03.874 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:03.875 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-20T19:34:03.876 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:03.952 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-dashboard 2024-09-20T19:34:04.354 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:55 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:04.488 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:04.488 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-20T19:34:04.489 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:04.489 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:34:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 145 k 2024-09-20T19:34:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:04.489 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:34:04.489 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:04.489 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-20T19:34:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:04.490 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 145 k 2024-09-20T19:34:04.490 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 436 k 2024-09-20T19:34:04.490 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:34:04.531 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:59 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:04.667 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:04.668 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:04.668 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-20T19:34:04.668 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:04.669 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:34:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 3.6 M 2024-09-20T19:34:04.669 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-20T19:34:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-20T19:34:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 32 k 2024-09-20T19:34:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 16 k 2024-09-20T19:34:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-20T19:34:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-20T19:34:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-20T19:34:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-20T19:34:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-20T19:34:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-20T19:34:04.669 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-20T19:34:04.670 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-20T19:34:04.670 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:04.670 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:34:04.670 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:04.670 INFO:teuthology.orchestra.run.smithi160.stdout:Install 12 Packages 2024-09-20T19:34:04.670 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:04.670 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 8.7 M 2024-09-20T19:34:04.670 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 105 M 2024-09-20T19:34:04.671 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:34:04.684 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-immutable-object-cache-19.1.1-369.ged8ee0f 750 kB/s | 145 kB 00:00 2024-09-20T19:34:04.684 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:04.685 INFO:teuthology.orchestra.run.smithi114.stdout:Total 744 kB/s | 145 kB 00:00 2024-09-20T19:34:04.685 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:34:04.692 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:34:04.692 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:34:04.750 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:34:04.750 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:34:04.914 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:34:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-369.ged8ee0f2.e 1/1 2024-09-20T19:34:04.993 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-369.ged8ee0f2.e 1/1 2024-09-20T19:34:04.993 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T19:34:04.993 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-20T19:34:04.993 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:05.314 INFO:teuthology.orchestra.run.smithi160.stdout:(1/12): ceph-prometheus-alerts-19.1.1-369.ged8e 110 kB/s | 16 kB 00:00 2024-09-20T19:34:05.339 INFO:teuthology.orchestra.run.smithi160.stdout:(2/12): ceph-grafana-dashboards-19.1.1-369.ged8 182 kB/s | 32 kB 00:00 2024-09-20T19:34:05.473 INFO:teuthology.orchestra.run.smithi160.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 269 kB 00:00 2024-09-20T19:34:05.540 INFO:teuthology.orchestra.run.smithi160.stdout:(4/12): ceph-mgr-dashboard-19.1.1-369.ged8ee0f2 9.5 MB/s | 3.6 MB 00:00 2024-09-20T19:34:05.574 INFO:teuthology.orchestra.run.smithi160.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.9 MB/s | 1.0 MB 00:00 2024-09-20T19:34:05.623 INFO:teuthology.orchestra.run.smithi160.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 6.4 MB/s | 548 kB 00:00 2024-09-20T19:34:05.649 INFO:teuthology.orchestra.run.smithi160.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 286 kB/s | 19 kB 00:00 2024-09-20T19:34:05.682 INFO:teuthology.orchestra.run.smithi160.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-09-20T19:34:05.708 INFO:teuthology.orchestra.run.smithi160.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-20T19:34:05.741 INFO:teuthology.orchestra.run.smithi160.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.6 MB/s | 188 kB 00:00 2024-09-20T19:34:05.775 INFO:teuthology.orchestra.run.smithi160.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.8 MB/s | 863 kB 00:00 2024-09-20T19:34:05.817 INFO:teuthology.orchestra.run.smithi160.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2024-09-20T19:34:05.820 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:05.821 INFO:teuthology.orchestra.run.smithi160.stdout:Total 7.5 MB/s | 8.7 MB 00:01 2024-09-20T19:34:05.869 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:34:05.886 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:34:05.886 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:34:05.921 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-369.ged8ee0f2.e 1/1 2024-09-20T19:34:05.921 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:05.921 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:34:05.921 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:05.921 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:05.921 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:06.098 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:34:06.098 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:34:06.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr 2024-09-20T19:34:06.624 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:34:06.709 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-20T19:34:06.748 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:57 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:06.752 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-20T19:34:06.799 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-20T19:34:06.830 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-20T19:34:06.841 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-mgr-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:06.885 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:06.886 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-20T19:34:06.886 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:06.936 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-20T19:34:06.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-dashboard 2024-09-20T19:34:07.071 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-20T19:34:07.165 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-20T19:34:07.217 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-20T19:34:07.254 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-20T19:34:07.286 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-369.ged8ee0f2.el9. 10/12 2024-09-20T19:34:07.530 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:58 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:07.661 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:07.662 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:07.662 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-20T19:34:07.663 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:07.663 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:34:07.663 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 3.6 M 2024-09-20T19:34:07.663 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-20T19:34:07.663 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-20T19:34:07.663 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 32 k 2024-09-20T19:34:07.663 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 16 k 2024-09-20T19:34:07.663 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-20T19:34:07.663 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-20T19:34:07.663 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-20T19:34:07.663 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-20T19:34:07.663 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-20T19:34:07.663 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-20T19:34:07.663 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-20T19:34:07.664 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-20T19:34:07.664 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:07.664 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:34:07.664 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:07.664 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-09-20T19:34:07.664 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:07.664 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 8.7 M 2024-09-20T19:34:07.664 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 105 M 2024-09-20T19:34:07.665 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:34:08.037 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): ceph-grafana-dashboards-19.1.1-369.ged8 223 kB/s | 32 kB 00:00 2024-09-20T19:34:08.062 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): ceph-prometheus-alerts-19.1.1-369.ged8e 98 kB/s | 16 kB 00:00 2024-09-20T19:34:08.304 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): ceph-mgr-dashboard-19.1.1-369.ged8ee0f2 8.7 MB/s | 3.6 MB 00:00 2024-09-20T19:34:08.329 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.0 MB/s | 269 kB 00:00 2024-09-20T19:34:08.342 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-369.ged8ee0f2.el9 11/12 2024-09-20T19:34:08.379 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-369.ged8ee0f2.el9.noar 12/12 2024-09-20T19:34:08.379 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-20T19:34:08.405 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 781 kB/s | 19 kB 00:00 2024-09-20T19:34:08.447 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.5 MB/s | 1.0 MB 00:00 2024-09-20T19:34:08.522 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 17 MB/s | 2.0 MB 00:00 2024-09-20T19:34:08.547 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-20T19:34:08.572 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.1 MB/s | 144 kB 00:00 2024-09-20T19:34:08.597 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.9 MB/s | 863 kB 00:00 2024-09-20T19:34:08.623 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-09-20T19:34:08.626 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:08.626 INFO:teuthology.orchestra.run.smithi114.stdout:Total 9.0 MB/s | 8.7 MB 00:00 2024-09-20T19:34:08.674 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:34:08.691 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:34:08.691 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:34:08.894 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:34:08.895 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:34:09.425 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:34:09.515 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-20T19:34:09.559 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-20T19:34:09.605 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-20T19:34:09.636 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-20T19:34:09.707 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-369.ged8ee0f2.el9.noar 12/12 2024-09-20T19:34:09.708 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-369.ged8ee0f2.el9 1/12 2024-09-20T19:34:09.708 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-369.ged8ee0f2.el9.noar 2/12 2024-09-20T19:34:09.708 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-369.ged8ee0f2.el9. 3/12 2024-09-20T19:34:09.708 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-20T19:34:09.708 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-20T19:34:09.708 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-20T19:34:09.708 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-20T19:34:09.708 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-20T19:34:09.708 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-20T19:34:09.708 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-20T19:34:09.708 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-20T19:34:09.736 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-20T19:34:09.862 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-20T19:34:09.956 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-20T19:34:10.008 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-20T19:34:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-20T19:34:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:10.023 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:34:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-20T19:34:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:34:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:34:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:34:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-20T19:34:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-20T19:34:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-20T19:34:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-20T19:34:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-20T19:34:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-20T19:34:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-20T19:34:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-20T19:34:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:10.024 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:10.035 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-20T19:34:10.067 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-369.ged8ee0f2.el9. 10/12 2024-09-20T19:34:10.286 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-20T19:34:10.871 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:05 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:11.007 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:11.008 INFO:teuthology.orchestra.run.smithi160.stdout:====================================================================================== 2024-09-20T19:34:11.008 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-20T19:34:11.008 INFO:teuthology.orchestra.run.smithi160.stdout:====================================================================================== 2024-09-20T19:34:11.008 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:34:11.008 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 7.4 M 2024-09-20T19:34:11.008 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-20T19:34:11.008 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-20T19:34:11.008 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-20T19:34:11.008 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-20T19:34:11.009 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-20T19:34:11.009 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-20T19:34:11.009 INFO:teuthology.orchestra.run.smithi160.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-20T19:34:11.009 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-20T19:34:11.009 INFO:teuthology.orchestra.run.smithi160.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-20T19:34:11.009 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-20T19:34:11.009 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-20T19:34:11.009 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-20T19:34:11.009 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:11.009 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:34:11.009 INFO:teuthology.orchestra.run.smithi160.stdout:====================================================================================== 2024-09-20T19:34:11.009 INFO:teuthology.orchestra.run.smithi160.stdout:Install 12 Packages 2024-09-20T19:34:11.009 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:11.010 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 38 M 2024-09-20T19:34:11.010 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 212 M 2024-09-20T19:34:11.010 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:34:11.127 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-369.ged8ee0f2.el9 11/12 2024-09-20T19:34:11.168 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-369.ged8ee0f2.el9.noar 12/12 2024-09-20T19:34:11.399 INFO:teuthology.orchestra.run.smithi160.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 595 kB/s | 191 kB 00:00 2024-09-20T19:34:11.425 INFO:teuthology.orchestra.run.smithi160.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.3 MB/s | 801 kB 00:00 2024-09-20T19:34:11.541 INFO:teuthology.orchestra.run.smithi160.stdout:(3/12): ceph-mgr-diskprediction-local-19.1.1-36 16 MB/s | 7.4 MB 00:00 2024-09-20T19:34:12.017 INFO:teuthology.orchestra.run.smithi160.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 53 kB/s | 33 kB 00:00 2024-09-20T19:34:12.042 INFO:teuthology.orchestra.run.smithi160.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 36 kB/s | 18 kB 00:00 2024-09-20T19:34:12.561 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-369.ged8ee0f2.el9.noar 12/12 2024-09-20T19:34:12.561 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-369.ged8ee0f2.el9 1/12 2024-09-20T19:34:12.561 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-369.ged8ee0f2.el9.noar 2/12 2024-09-20T19:34:12.561 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-369.ged8ee0f2.el9. 3/12 2024-09-20T19:34:12.561 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-20T19:34:12.561 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-20T19:34:12.561 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-20T19:34:12.561 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-20T19:34:12.561 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-20T19:34:12.561 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-20T19:34:12.562 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-20T19:34:12.562 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-20T19:34:12.576 INFO:teuthology.orchestra.run.smithi160.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-09-20T19:34:12.618 INFO:teuthology.orchestra.run.smithi160.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 64 kB/s | 38 kB 00:00 2024-09-20T19:34:12.897 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-20T19:34:12.898 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:12.898 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:34:12.898 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-20T19:34:12.898 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:34:12.898 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:34:12.898 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:34:12.898 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-20T19:34:12.898 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-20T19:34:12.898 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-20T19:34:12.898 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-20T19:34:12.898 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-20T19:34:12.899 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-20T19:34:12.899 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-20T19:34:12.899 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-20T19:34:12.899 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:12.899 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:13.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-20T19:34:13.159 INFO:teuthology.orchestra.run.smithi160.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 4.4 MB/s | 4.9 MB 00:01 2024-09-20T19:34:13.335 INFO:teuthology.orchestra.run.smithi160.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 323 kB/s | 245 kB 00:00 2024-09-20T19:34:13.715 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:04 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:13.845 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:13.846 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-09-20T19:34:13.846 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-20T19:34:13.846 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-09-20T19:34:13.846 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:34:13.846 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 7.4 M 2024-09-20T19:34:13.846 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-20T19:34:13.846 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-20T19:34:13.846 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-20T19:34:13.846 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-20T19:34:13.847 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-20T19:34:13.847 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-20T19:34:13.847 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-20T19:34:13.847 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-20T19:34:13.847 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-20T19:34:13.847 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-20T19:34:13.847 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-20T19:34:13.847 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-20T19:34:13.847 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:13.847 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:34:13.847 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-09-20T19:34:13.847 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-09-20T19:34:13.847 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:13.848 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 38 M 2024-09-20T19:34:13.848 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 212 M 2024-09-20T19:34:13.848 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:34:13.893 INFO:teuthology.orchestra.run.smithi160.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.0 MB/s | 5.1 MB 00:01 2024-09-20T19:34:13.960 INFO:teuthology.orchestra.run.smithi160.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 313 kB/s | 250 kB 00:00 2024-09-20T19:34:14.122 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-20T19:34:14.164 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 788 kB/s | 33 kB 00:00 2024-09-20T19:34:14.189 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.0 MB/s | 801 kB 00:00 2024-09-20T19:34:14.231 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 439 kB/s | 18 kB 00:00 2024-09-20T19:34:14.273 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 926 kB/s | 38 kB 00:00 2024-09-20T19:34:14.473 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): ceph-mgr-diskprediction-local-19.1.1-36 15 MB/s | 7.4 MB 00:00 2024-09-20T19:34:14.590 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-09-20T19:34:14.666 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.0 MB/s | 3.0 MB 00:00 2024-09-20T19:34:14.708 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.8 MB/s | 250 kB 00:00 2024-09-20T19:34:14.777 INFO:teuthology.orchestra.run.smithi160.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-20T19:34:14.781 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:14.781 INFO:teuthology.orchestra.run.smithi160.stdout:Total 10 MB/s | 38 MB 00:03 2024-09-20T19:34:15.050 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-09-20T19:34:15.133 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:34:15.160 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:34:15.160 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:34:15.608 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:34:15.608 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:34:15.616 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.7 MB/s | 4.9 MB 00:01 2024-09-20T19:34:15.944 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.8 MB/s | 5.1 MB 00:01 2024-09-20T19:34:15.944 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:15.944 INFO:teuthology.orchestra.run.smithi114.stdout:Total 18 MB/s | 38 MB 00:02 2024-09-20T19:34:16.273 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:34:16.298 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:34:16.298 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:34:16.390 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:34:16.473 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-20T19:34:16.518 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-20T19:34:16.620 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-20T19:34:16.660 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-20T19:34:16.715 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:34:16.715 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:34:16.857 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-20T19:34:16.901 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-20T19:34:16.979 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-20T19:34:17.473 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:34:17.543 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-20T19:34:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-20T19:34:17.639 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-20T19:34:17.680 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-20T19:34:17.716 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-20T19:34:17.765 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-20T19:34:17.899 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-20T19:34:17.931 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-20T19:34:18.017 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-20T19:34:18.655 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-20T19:34:18.771 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-20T19:34:19.028 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-20T19:34:20.000 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-20T19:34:20.059 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-20T19:34:20.091 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-369.ged8ee0 12/12 2024-09-20T19:34:21.027 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-20T19:34:21.050 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-369.ged8ee0 12/12 2024-09-20T19:34:22.747 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-369.ged8ee0 12/12 2024-09-20T19:34:22.748 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-369.ged8ee0 1/12 2024-09-20T19:34:22.748 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-20T19:34:22.748 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-20T19:34:22.748 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-20T19:34:22.748 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-20T19:34:22.748 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-20T19:34:22.748 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-20T19:34:22.748 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-20T19:34:22.748 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-20T19:34:22.748 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-20T19:34:22.748 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-20T19:34:23.124 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-20T19:34:23.124 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:23.124 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:34:23.124 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:34:23.124 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-20T19:34:23.125 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-20T19:34:23.125 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-20T19:34:23.125 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-20T19:34:23.125 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-20T19:34:23.125 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-20T19:34:23.125 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-20T19:34:23.125 INFO:teuthology.orchestra.run.smithi160.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-20T19:34:23.125 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-20T19:34:23.125 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-20T19:34:23.125 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-20T19:34:23.125 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:23.125 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:23.538 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-rook 2024-09-20T19:34:23.818 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-369.ged8ee0 12/12 2024-09-20T19:34:23.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-369.ged8ee0 1/12 2024-09-20T19:34:23.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-20T19:34:23.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-20T19:34:23.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-20T19:34:23.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-20T19:34:23.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-20T19:34:23.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-20T19:34:23.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-20T19:34:23.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-20T19:34:23.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-20T19:34:23.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-20T19:34:24.124 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:19 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:24.240 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-20T19:34:24.240 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:24.240 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:34:24.240 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:34:24.240 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-20T19:34:24.240 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-20T19:34:24.240 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-20T19:34:24.240 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-20T19:34:24.240 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-20T19:34:24.240 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-20T19:34:24.241 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-20T19:34:24.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-20T19:34:24.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-20T19:34:24.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-20T19:34:24.241 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-20T19:34:24.241 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:24.241 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:24.256 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:24.257 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-20T19:34:24.257 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:24.257 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:34:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 49 k 2024-09-20T19:34:24.257 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-20T19:34:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-20T19:34:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-20T19:34:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-20T19:34:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-20T19:34:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-20T19:34:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-20T19:34:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-20T19:34:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-20T19:34:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-20T19:34:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-20T19:34:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-20T19:34:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-20T19:34:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:24.258 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:34:24.258 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:24.258 INFO:teuthology.orchestra.run.smithi160.stdout:Install 13 Packages 2024-09-20T19:34:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:24.259 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 2.2 M 2024-09-20T19:34:24.259 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 26 M 2024-09-20T19:34:24.259 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:34:24.508 INFO:teuthology.orchestra.run.smithi160.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 152 kB/s | 26 kB 00:00 2024-09-20T19:34:24.533 INFO:teuthology.orchestra.run.smithi160.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 99 kB/s | 19 kB 00:00 2024-09-20T19:34:24.558 INFO:teuthology.orchestra.run.smithi160.stdout:(3/13): ceph-mgr-rook-19.1.1-369.ged8ee0f2.el9. 225 kB/s | 49 kB 00:00 2024-09-20T19:34:24.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-rook 2024-09-20T19:34:24.684 INFO:teuthology.orchestra.run.smithi160.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 222 kB 00:00 2024-09-20T19:34:24.718 INFO:teuthology.orchestra.run.smithi160.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 863 kB/s | 159 kB 00:00 2024-09-20T19:34:24.751 INFO:teuthology.orchestra.run.smithi160.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 801 kB/s | 54 kB 00:00 2024-09-20T19:34:24.784 INFO:teuthology.orchestra.run.smithi160.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 483 kB/s | 32 kB 00:00 2024-09-20T19:34:24.810 INFO:teuthology.orchestra.run.smithi160.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 248 kB/s | 14 kB 00:00 2024-09-20T19:34:24.843 INFO:teuthology.orchestra.run.smithi160.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 980 kB/s | 279 kB 00:00 2024-09-20T19:34:24.877 INFO:teuthology.orchestra.run.smithi160.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.4 MB/s | 220 kB 00:00 2024-09-20T19:34:24.902 INFO:teuthology.orchestra.run.smithi160.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-20T19:34:24.927 INFO:teuthology.orchestra.run.smithi160.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 706 kB/s | 59 kB 00:00 2024-09-20T19:34:24.961 INFO:teuthology.orchestra.run.smithi160.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-20T19:34:24.964 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:24.964 INFO:teuthology.orchestra.run.smithi160.stdout:Total 3.1 MB/s | 2.2 MB 00:00 2024-09-20T19:34:25.030 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:34:25.044 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:34:25.044 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:34:25.136 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:34:25.136 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:34:25.179 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:16 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:25.313 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:25.314 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:25.314 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-20T19:34:25.314 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:25.314 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:34:25.314 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 49 k 2024-09-20T19:34:25.314 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-20T19:34:25.314 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-20T19:34:25.314 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-20T19:34:25.314 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-20T19:34:25.314 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-20T19:34:25.314 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-20T19:34:25.314 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-20T19:34:25.315 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-20T19:34:25.315 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-20T19:34:25.315 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-20T19:34:25.315 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-20T19:34:25.315 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-20T19:34:25.315 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-20T19:34:25.315 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:25.315 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:34:25.315 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:25.315 INFO:teuthology.orchestra.run.smithi114.stdout:Install 13 Packages 2024-09-20T19:34:25.315 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:25.316 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 2.2 M 2024-09-20T19:34:25.316 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 26 M 2024-09-20T19:34:25.316 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:34:25.509 INFO:teuthology.orchestra.run.smithi114.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 228 kB/s | 26 kB 00:00 2024-09-20T19:34:25.534 INFO:teuthology.orchestra.run.smithi114.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 139 kB/s | 19 kB 00:00 2024-09-20T19:34:25.562 INFO:teuthology.orchestra.run.smithi114.stdout:(3/13): ceph-mgr-rook-19.1.1-369.ged8ee0f2.el9. 302 kB/s | 49 kB 00:00 2024-09-20T19:34:25.667 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:34:25.676 INFO:teuthology.orchestra.run.smithi114.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-20T19:34:25.702 INFO:teuthology.orchestra.run.smithi114.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 950 kB/s | 159 kB 00:00 2024-09-20T19:34:25.735 INFO:teuthology.orchestra.run.smithi114.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 915 kB/s | 54 kB 00:00 2024-09-20T19:34:25.769 INFO:teuthology.orchestra.run.smithi114.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-20T19:34:25.794 INFO:teuthology.orchestra.run.smithi114.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-20T19:34:25.815 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-20T19:34:25.819 INFO:teuthology.orchestra.run.smithi114.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 172 kB/s | 14 kB 00:00 2024-09-20T19:34:25.870 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-20T19:34:25.878 INFO:teuthology.orchestra.run.smithi114.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-20T19:34:25.903 INFO:teuthology.orchestra.run.smithi114.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 770 kB/s | 59 kB 00:00 2024-09-20T19:34:25.925 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-20T19:34:25.928 INFO:teuthology.orchestra.run.smithi114.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-20T19:34:25.966 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-20T19:34:26.013 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-20T19:34:26.020 INFO:teuthology.orchestra.run.smithi114.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.5 MB/s | 1.0 MB 00:00 2024-09-20T19:34:26.023 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:26.023 INFO:teuthology.orchestra.run.smithi114.stdout:Total 3.1 MB/s | 2.2 MB 00:00 2024-09-20T19:34:26.086 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:34:26.101 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:34:26.101 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:34:26.139 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-20T19:34:26.192 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:34:26.193 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:34:26.241 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-20T19:34:26.298 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-20T19:34:26.707 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:34:26.873 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-20T19:34:26.930 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-20T19:34:26.993 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-20T19:34:27.020 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-20T19:34:27.039 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-20T19:34:27.070 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-20T19:34:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-20T19:34:27.103 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-20T19:34:27.151 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-20T19:34:27.182 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-rook-2:19.1.1-369.ged8ee0f2.el9.noarch 13/13 2024-09-20T19:34:27.190 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-20T19:34:27.291 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-20T19:34:27.341 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-20T19:34:27.968 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-369.ged8ee0f2.el9.noarch 13/13 2024-09-20T19:34:27.968 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-rook-2:19.1.1-369.ged8ee0f2.el9.noarch 1/13 2024-09-20T19:34:27.968 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-20T19:34:27.968 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-20T19:34:27.968 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-20T19:34:27.968 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-20T19:34:27.969 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-20T19:34:27.969 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-20T19:34:27.969 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-20T19:34:27.969 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-20T19:34:27.969 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-20T19:34:27.969 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-20T19:34:27.969 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-20T19:34:28.058 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-20T19:34:28.105 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-20T19:34:28.139 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-20T19:34:28.188 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-20T19:34:28.217 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:19.1.1-369.ged8ee0f2.el9.noarch 13/13 2024-09-20T19:34:28.304 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-20T19:34:28.304 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:28.304 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:34:28.304 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:34:28.304 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-20T19:34:28.304 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-20T19:34:28.304 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-20T19:34:28.304 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-20T19:34:28.304 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-20T19:34:28.304 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-20T19:34:28.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-20T19:34:28.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-20T19:34:28.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-20T19:34:28.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-20T19:34:28.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-20T19:34:28.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-20T19:34:28.305 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:28.305 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:28.609 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-cephadm 2024-09-20T19:34:28.956 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-369.ged8ee0f2.el9.noarch 13/13 2024-09-20T19:34:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:19.1.1-369.ged8ee0f2.el9.noarch 1/13 2024-09-20T19:34:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-20T19:34:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-20T19:34:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-20T19:34:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-20T19:34:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-20T19:34:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-20T19:34:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-20T19:34:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-20T19:34:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-20T19:34:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-20T19:34:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-20T19:34:29.196 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:24 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:29.295 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-20T19:34:29.295 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:29.295 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:34:29.295 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:34:29.295 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-20T19:34:29.295 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-20T19:34:29.295 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-20T19:34:29.295 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-20T19:34:29.295 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-20T19:34:29.295 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-20T19:34:29.295 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-20T19:34:29.295 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-20T19:34:29.296 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-20T19:34:29.296 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-20T19:34:29.296 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-20T19:34:29.296 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-20T19:34:29.296 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:29.296 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:29.328 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:29.329 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:29.329 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-20T19:34:29.329 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:29.329 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:34:29.329 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 146 k 2024-09-20T19:34:29.329 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-20T19:34:29.329 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-20T19:34:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-20T19:34:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-20T19:34:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-20T19:34:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-20T19:34:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:29.330 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:34:29.330 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:29.330 INFO:teuthology.orchestra.run.smithi160.stdout:Install 6 Packages 2024-09-20T19:34:29.330 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:29.331 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 7.0 M 2024-09-20T19:34:29.331 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 33 M 2024-09-20T19:34:29.331 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:34:29.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-cephadm 2024-09-20T19:34:29.838 INFO:teuthology.orchestra.run.smithi160.stdout:(1/6): ceph-mgr-cephadm-19.1.1-369.ged8ee0f2.el 759 kB/s | 146 kB 00:00 2024-09-20T19:34:29.863 INFO:teuthology.orchestra.run.smithi160.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-20T19:34:29.921 INFO:teuthology.orchestra.run.smithi160.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-20T19:34:29.947 INFO:teuthology.orchestra.run.smithi160.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 690 kB/s | 58 kB 00:00 2024-09-20T19:34:29.972 INFO:teuthology.orchestra.run.smithi160.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-20T19:34:30.139 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:21 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:30.271 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:30.271 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:30.271 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-20T19:34:30.272 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:30.272 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:34:30.272 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:19.1.1-369.ged8ee0f2.el9 ceph-noarch 146 k 2024-09-20T19:34:30.272 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-20T19:34:30.272 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-20T19:34:30.272 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-20T19:34:30.272 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-20T19:34:30.272 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-20T19:34:30.272 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-20T19:34:30.272 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:30.272 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:34:30.272 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:30.273 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-09-20T19:34:30.273 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:30.273 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 7.0 M 2024-09-20T19:34:30.273 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 33 M 2024-09-20T19:34:30.273 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:34:30.639 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): ceph-mgr-cephadm-19.1.1-369.ged8ee0f2.el 662 kB/s | 146 kB 00:00 2024-09-20T19:34:30.706 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.9 MB/s | 548 kB 00:00 2024-09-20T19:34:30.732 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-20T19:34:30.757 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.0 MB/s | 74 kB 00:00 2024-09-20T19:34:31.240 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 302 kB/s | 248 kB 00:00 2024-09-20T19:34:31.322 INFO:teuthology.orchestra.run.smithi160.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.6 MB/s | 6.0 MB 00:01 2024-09-20T19:34:31.324 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:31.325 INFO:teuthology.orchestra.run.smithi160.stdout:Total 3.5 MB/s | 7.0 MB 00:01 2024-09-20T19:34:31.396 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:34:31.408 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:34:31.408 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:34:31.508 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:34:31.508 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:34:31.749 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.5 MB/s | 6.0 MB 00:01 2024-09-20T19:34:31.752 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:31.752 INFO:teuthology.orchestra.run.smithi114.stdout:Total 4.7 MB/s | 7.0 MB 00:01 2024-09-20T19:34:31.819 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:34:31.824 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:34:31.836 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:34:31.836 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:34:31.939 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:34:31.940 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:34:31.989 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-20T19:34:32.049 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-20T19:34:32.252 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:34:32.431 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-20T19:34:32.484 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-20T19:34:32.574 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-20T19:34:32.659 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-20T19:34:32.724 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-20T19:34:32.753 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 6/6 2024-09-20T19:34:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-20T19:34:33.112 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-20T19:34:33.176 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-20T19:34:33.230 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 6/6 2024-09-20T19:34:33.496 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 6/6 2024-09-20T19:34:33.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 1/6 2024-09-20T19:34:33.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-20T19:34:33.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-20T19:34:33.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-20T19:34:33.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-20T19:34:33.811 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-20T19:34:33.811 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:33.812 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:34:33.812 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:34:33.812 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-20T19:34:33.812 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-20T19:34:33.812 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-20T19:34:33.812 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-20T19:34:33.812 INFO:teuthology.orchestra.run.smithi160.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-20T19:34:33.812 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:33.812 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 6/6 2024-09-20T19:34:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 1/6 2024-09-20T19:34:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-20T19:34:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-20T19:34:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-20T19:34:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-20T19:34:34.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-fuse 2024-09-20T19:34:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-20T19:34:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:34.302 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:34:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:19.1.1-369.ged8ee0f2.el9.noarch 2024-09-20T19:34:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-20T19:34:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-20T19:34:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-20T19:34:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-20T19:34:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-20T19:34:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:34.303 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:34.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-fuse 2024-09-20T19:34:34.669 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:29 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:34.802 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:34.803 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:34.803 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-20T19:34:34.803 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:34.803 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:34:34.803 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 897 k 2024-09-20T19:34:34.803 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-20T19:34:34.803 INFO:teuthology.orchestra.run.smithi160.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-20T19:34:34.804 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:34.804 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:34:34.804 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:34.804 INFO:teuthology.orchestra.run.smithi160.stdout:Install 2 Packages 2024-09-20T19:34:34.804 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:34.804 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 976 k 2024-09-20T19:34:34.804 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 2.9 M 2024-09-20T19:34:34.805 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:34:35.132 INFO:teuthology.orchestra.run.smithi160.stdout:(1/2): ceph-fuse-19.1.1-369.ged8ee0f2.el9.x86_6 3.1 MB/s | 897 kB 00:00 2024-09-20T19:34:35.152 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:26 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:35.283 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:35.284 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:35.284 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-20T19:34:35.284 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:35.284 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:34:35.284 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 897 k 2024-09-20T19:34:35.284 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-20T19:34:35.284 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-20T19:34:35.284 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:35.284 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:34:35.284 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:35.284 INFO:teuthology.orchestra.run.smithi114.stdout:Install 2 Packages 2024-09-20T19:34:35.284 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:35.285 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 976 k 2024-09-20T19:34:35.285 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 2.9 M 2024-09-20T19:34:35.285 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:34:35.518 INFO:teuthology.orchestra.run.smithi114.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 694 kB/s | 80 kB 00:00 2024-09-20T19:34:35.626 INFO:teuthology.orchestra.run.smithi160.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 103 kB/s | 80 kB 00:00 2024-09-20T19:34:35.626 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:35.626 INFO:teuthology.orchestra.run.smithi160.stdout:Total 1.2 MB/s | 976 kB 00:00 2024-09-20T19:34:35.638 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:34:35.662 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:34:35.662 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:34:35.668 INFO:teuthology.orchestra.run.smithi114.stdout:(2/2): ceph-fuse-19.1.1-369.ged8ee0f2.el9.x86_6 3.3 MB/s | 897 kB 00:00 2024-09-20T19:34:35.668 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:35.669 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.5 MB/s | 976 kB 00:00 2024-09-20T19:34:35.681 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:34:35.702 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:34:35.702 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:34:35.751 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:34:35.751 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:34:35.788 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:34:35.788 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:34:35.939 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:34:35.989 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:34:36.089 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-20T19:34:36.146 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-20T19:34:36.165 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 2/2 2024-09-20T19:34:36.217 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 2/2 2024-09-20T19:34:36.777 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 2/2 2024-09-20T19:34:36.777 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/2 2024-09-20T19:34:36.904 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 2/2 2024-09-20T19:34:36.904 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/2 2024-09-20T19:34:37.046 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-20T19:34:37.046 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:37.046 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:34:37.046 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-20T19:34:37.046 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:37.046 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:37.186 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-20T19:34:37.186 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:37.186 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:34:37.186 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-20T19:34:37.186 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:37.186 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:37.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados-devel 2024-09-20T19:34:37.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados-devel 2024-09-20T19:34:37.953 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:32 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:38.060 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:29 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:38.089 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:38.089 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:38.090 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-09-20T19:34:38.090 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:38.090 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:34:38.090 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 126 k 2024-09-20T19:34:38.090 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:38.090 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:34:38.090 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:38.090 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-20T19:34:38.090 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:38.091 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 126 k 2024-09-20T19:34:38.091 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 456 k 2024-09-20T19:34:38.091 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:34:38.193 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:38.194 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:38.194 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-20T19:34:38.194 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:38.194 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:34:38.194 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 126 k 2024-09-20T19:34:38.194 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:38.194 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:34:38.194 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:38.194 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-20T19:34:38.194 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:38.194 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 126 k 2024-09-20T19:34:38.194 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 456 k 2024-09-20T19:34:38.194 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:34:38.303 INFO:teuthology.orchestra.run.smithi160.stdout:librados-devel-19.1.1-369.ged8ee0f2.el9.x86_64. 598 kB/s | 126 kB 00:00 2024-09-20T19:34:38.303 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:38.304 INFO:teuthology.orchestra.run.smithi160.stdout:Total 593 kB/s | 126 kB 00:00 2024-09-20T19:34:38.304 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:34:38.309 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:34:38.309 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:34:38.372 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:34:38.373 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:34:38.405 INFO:teuthology.orchestra.run.smithi114.stdout:librados-devel-19.1.1-369.ged8ee0f2.el9.x86_64. 601 kB/s | 126 kB 00:00 2024-09-20T19:34:38.406 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:38.406 INFO:teuthology.orchestra.run.smithi114.stdout:Total 596 kB/s | 126 kB 00:00 2024-09-20T19:34:38.407 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:34:38.412 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:34:38.412 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:34:38.475 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:34:38.475 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:34:38.523 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:34:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librados-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:38.634 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:34:38.737 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:39.238 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:39.310 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:39.310 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:39.310 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:34:39.310 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:39.310 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:39.310 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:39.494 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:39.494 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:39.494 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:34:39.494 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:39.494 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:39.494 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:39.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs2 2024-09-20T19:34:39.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs2 2024-09-20T19:34:40.149 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:35 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:40.240 INFO:teuthology.orchestra.run.smithi160.stdout:Package libcephfs2-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:40.284 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:40.285 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-20T19:34:40.285 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:40.346 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:31 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:40.366 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs-devel 2024-09-20T19:34:40.435 INFO:teuthology.orchestra.run.smithi114.stdout:Package libcephfs2-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:40.480 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:40.481 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-20T19:34:40.481 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:40.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs-devel 2024-09-20T19:34:40.952 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:35 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:41.084 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:41.084 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:41.085 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-09-20T19:34:41.085 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:41.085 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:34:41.085 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 32 k 2024-09-20T19:34:41.085 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:41.085 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:34:41.085 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:41.085 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-20T19:34:41.085 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:41.086 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 32 k 2024-09-20T19:34:41.086 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 142 k 2024-09-20T19:34:41.086 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:34:41.133 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:32 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:41.239 INFO:teuthology.orchestra.run.smithi160.stdout:libcephfs-devel-19.1.1-369.ged8ee0f2.el9.x86_64 207 kB/s | 32 kB 00:00 2024-09-20T19:34:41.240 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:41.240 INFO:teuthology.orchestra.run.smithi160.stdout:Total 205 kB/s | 32 kB 00:00 2024-09-20T19:34:41.240 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:34:41.243 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:34:41.244 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:34:41.264 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:41.265 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:41.265 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-20T19:34:41.265 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:41.265 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:34:41.265 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 32 k 2024-09-20T19:34:41.265 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:41.265 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:34:41.265 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:41.265 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-20T19:34:41.265 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:41.266 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:34:41.266 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:34:41.266 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 32 k 2024-09-20T19:34:41.267 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 142 k 2024-09-20T19:34:41.267 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:34:41.355 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:34:41.418 INFO:teuthology.orchestra.run.smithi114.stdout:libcephfs-devel-19.1.1-369.ged8ee0f2.el9.x86_64 209 kB/s | 32 kB 00:00 2024-09-20T19:34:41.419 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:41.419 INFO:teuthology.orchestra.run.smithi114.stdout:Total 207 kB/s | 32 kB 00:00 2024-09-20T19:34:41.419 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:34:41.422 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:34:41.422 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:34:41.442 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:34:41.442 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:34:41.463 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:41.524 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:34:41.640 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:41.833 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:42.029 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:42.058 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:42.058 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:42.058 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:34:42.058 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:42.058 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:42.058 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:42.275 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:42.275 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:42.275 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:34:42.275 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:42.275 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:42.275 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:42.286 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados2 2024-09-20T19:34:42.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados2 2024-09-20T19:34:42.867 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:37 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:42.958 INFO:teuthology.orchestra.run.smithi160.stdout:Package librados2-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:43.000 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:43.001 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-20T19:34:43.001 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:43.031 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:34 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:43.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librbd1 2024-09-20T19:34:43.119 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:43.162 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:43.163 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-20T19:34:43.163 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:43.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librbd1 2024-09-20T19:34:43.668 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:38 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:43.756 INFO:teuthology.orchestra.run.smithi160.stdout:Package librbd1-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:43.799 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:43.800 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-20T19:34:43.800 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:43.816 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:34 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:43.882 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rados 2024-09-20T19:34:43.906 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:43.949 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:43.950 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-20T19:34:43.950 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:44.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rados 2024-09-20T19:34:44.471 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:39 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:44.562 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rados-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:44.604 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:35 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:44.605 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:44.607 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-20T19:34:44.607 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:44.687 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rgw 2024-09-20T19:34:44.692 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rados-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:44.736 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:44.737 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-20T19:34:44.737 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:44.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rgw 2024-09-20T19:34:45.289 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:40 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:45.383 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:45.388 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:36 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:45.430 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:45.431 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-20T19:34:45.431 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:45.479 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rgw-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:45.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-cephfs 2024-09-20T19:34:45.522 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:45.523 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-20T19:34:45.523 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:45.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-cephfs 2024-09-20T19:34:46.110 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:41 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:46.173 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:37 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:46.201 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-cephfs-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:46.244 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:46.245 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-20T19:34:46.245 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:46.261 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-cephfs-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:46.304 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:46.305 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-20T19:34:46.305 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:46.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rbd 2024-09-20T19:34:46.364 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rbd 2024-09-20T19:34:46.914 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:41 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:46.948 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:37 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:47.004 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:47.038 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 is already installed. 2024-09-20T19:34:47.047 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:47.048 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-20T19:34:47.048 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:47.081 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:47.082 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-20T19:34:47.082 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:47.132 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-fuse 2024-09-20T19:34:47.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-fuse 2024-09-20T19:34:47.709 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:42 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:47.727 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:38 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:47.841 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:47.842 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:47.842 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-09-20T19:34:47.842 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:47.842 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:34:47.842 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 85 k 2024-09-20T19:34:47.842 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:47.842 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:34:47.842 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:47.842 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-20T19:34:47.842 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:47.842 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 85 k 2024-09-20T19:34:47.843 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 231 k 2024-09-20T19:34:47.843 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:34:47.859 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:47.860 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:47.861 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-20T19:34:47.861 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:47.861 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:34:47.861 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 85 k 2024-09-20T19:34:47.861 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:47.861 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:34:47.862 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:47.862 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-20T19:34:47.862 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:47.862 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 85 k 2024-09-20T19:34:47.862 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 231 k 2024-09-20T19:34:47.862 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:34:48.010 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-fuse-19.1.1-369.ged8ee0f2.el9.x86_64.rpm 511 kB/s | 85 kB 00:00 2024-09-20T19:34:48.010 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:48.010 INFO:teuthology.orchestra.run.smithi160.stdout:Total 506 kB/s | 85 kB 00:00 2024-09-20T19:34:48.011 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:34:48.017 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:34:48.017 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:34:48.046 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-fuse-19.1.1-369.ged8ee0f2.el9.x86_64.rpm 461 kB/s | 85 kB 00:00 2024-09-20T19:34:48.046 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:48.046 INFO:teuthology.orchestra.run.smithi114.stdout:Total 458 kB/s | 85 kB 00:00 2024-09-20T19:34:48.046 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:34:48.053 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:34:48.053 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:34:48.080 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:34:48.080 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:34:48.117 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:34:48.117 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:34:48.237 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:34:48.270 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:34:48.340 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:48.367 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:48.733 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:48.778 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:48.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:48.983 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:48.983 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:34:48.983 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:48.984 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:48.984 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:49.034 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:49.034 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:49.034 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:34:49.035 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:49.035 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:49.035 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:49.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-mirror 2024-09-20T19:34:49.285 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-mirror 2024-09-20T19:34:49.818 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:40 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:49.889 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:44 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:49.952 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:49.953 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:49.953 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-20T19:34:49.953 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:49.953 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:34:49.953 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 3.1 M 2024-09-20T19:34:49.953 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:49.953 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:34:49.953 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:49.953 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-20T19:34:49.953 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:49.953 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 3.1 M 2024-09-20T19:34:49.953 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 13 M 2024-09-20T19:34:49.954 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:34:50.027 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:50.028 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:50.028 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-09-20T19:34:50.028 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:50.028 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:34:50.028 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 3.1 M 2024-09-20T19:34:50.028 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:50.028 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:34:50.029 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:50.029 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-20T19:34:50.029 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:50.029 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 3.1 M 2024-09-20T19:34:50.029 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 13 M 2024-09-20T19:34:50.029 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:34:50.274 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-mirror-19.1.1-369.ged8ee0f2.el9.x86_64.rpm 9.7 MB/s | 3.1 MB 00:00 2024-09-20T19:34:50.275 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:50.275 INFO:teuthology.orchestra.run.smithi114.stdout:Total 9.6 MB/s | 3.1 MB 00:00 2024-09-20T19:34:50.275 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:34:50.284 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:34:50.284 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:34:50.365 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:34:50.366 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:34:50.371 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-mirror-19.1.1-369.ged8ee0f2.el9.x86_64.rpm 9.1 MB/s | 3.1 MB 00:00 2024-09-20T19:34:50.372 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:50.372 INFO:teuthology.orchestra.run.smithi160.stdout:Total 9.0 MB/s | 3.1 MB 00:00 2024-09-20T19:34:50.372 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:34:50.381 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:34:50.381 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:34:50.464 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:34:50.464 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:34:50.801 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:34:50.892 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:50.902 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:34:50.916 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:50.916 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T19:34:50.916 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-20T19:34:50.916 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-20T19:34:50.916 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-20T19:34:50.916 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:50.990 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-mirror-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:51.014 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-mirror-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:51.014 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T19:34:51.014 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-20T19:34:51.014 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-20T19:34:51.014 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-20T19:34:51.014 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:52.042 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:52.043 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:34:52.043 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:52.043 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:52.139 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-mirror-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:52.139 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:52.139 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:34:52.139 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:52.139 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:52.139 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:52.284 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-nbd 2024-09-20T19:34:52.409 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-nbd 2024-09-20T19:34:52.868 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:43 ago on Fri 20 Sep 2024 07:33:09 PM UTC. 2024-09-20T19:34:53.004 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-20T19:34:53.004 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:53.004 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-20T19:34:53.004 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:53.004 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-20T19:34:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 170 k 2024-09-20T19:34:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:53.005 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-20T19:34:53.005 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-20T19:34:53.005 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-20T19:34:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:53.005 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 170 k 2024-09-20T19:34:53.006 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 494 k 2024-09-20T19:34:53.006 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-20T19:34:53.008 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:48 ago on Fri 20 Sep 2024 07:33:05 PM UTC. 2024-09-20T19:34:53.142 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-20T19:34:53.142 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:53.142 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-09-20T19:34:53.142 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:53.142 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-20T19:34:53.143 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd x86_64 2:19.1.1-369.ged8ee0f2.el9 ceph 170 k 2024-09-20T19:34:53.143 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:53.143 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-20T19:34:53.143 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-20T19:34:53.143 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-20T19:34:53.143 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:53.143 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 170 k 2024-09-20T19:34:53.143 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 494 k 2024-09-20T19:34:53.144 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-20T19:34:53.220 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-nbd-19.1.1-369.ged8ee0f2.el9.x86_64.rpm 796 kB/s | 170 kB 00:00 2024-09-20T19:34:53.220 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:53.221 INFO:teuthology.orchestra.run.smithi114.stdout:Total 790 kB/s | 170 kB 00:00 2024-09-20T19:34:53.221 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-20T19:34:53.228 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-20T19:34:53.228 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-20T19:34:53.291 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-20T19:34:53.292 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-20T19:34:53.350 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-nbd-19.1.1-369.ged8ee0f2.el9.x86_64.rpm 826 kB/s | 170 kB 00:00 2024-09-20T19:34:53.351 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:34:53.351 INFO:teuthology.orchestra.run.smithi160.stdout:Total 820 kB/s | 170 kB 00:00 2024-09-20T19:34:53.351 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-20T19:34:53.359 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-20T19:34:53.359 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-20T19:34:53.423 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-20T19:34:53.423 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-20T19:34:53.452 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-20T19:34:53.551 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:53.574 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-20T19:34:53.681 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-nbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:53.980 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-nbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-nbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:54.254 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:54.254 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:54.254 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-20T19:34:54.254 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:54.254 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:34:54.254 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-20T19:34:54.376 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-nbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 1/1 2024-09-20T19:34:54.376 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:54.376 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-20T19:34:54.376 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd-2:19.1.1-369.ged8ee0f2.el9.x86_64 2024-09-20T19:34:54.376 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:34:54.377 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-20T19:34:54.557 DEBUG:teuthology.parallel:result is None 2024-09-20T19:34:54.640 DEBUG:teuthology.parallel:result is None 2024-09-20T19:34:54.640 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-20T19:34:54.641 INFO:teuthology.packaging:ref: None 2024-09-20T19:34:54.641 INFO:teuthology.packaging:tag: None 2024-09-20T19:34:54.641 INFO:teuthology.packaging:branch: squid 2024-09-20T19:34:54.641 INFO:teuthology.packaging:sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:34:54.641 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-20T19:34:54.886 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-20T19:34:54.909 INFO:teuthology.orchestra.run.smithi114.stdout:19.1.1-369.ged8ee0f2.el9 2024-09-20T19:34:54.909 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-369.ged8ee0f2.el9 2024-09-20T19:34:54.909 INFO:teuthology.task.install:The correct ceph version 19.1.1-369.ged8ee0f2 is installed. 2024-09-20T19:34:54.911 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-20T19:34:54.911 INFO:teuthology.packaging:ref: None 2024-09-20T19:34:54.911 INFO:teuthology.packaging:tag: None 2024-09-20T19:34:54.911 INFO:teuthology.packaging:branch: squid 2024-09-20T19:34:54.911 INFO:teuthology.packaging:sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:34:54.911 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-20T19:34:55.143 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-20T19:34:55.165 INFO:teuthology.orchestra.run.smithi160.stdout:19.1.1-369.ged8ee0f2.el9 2024-09-20T19:34:55.165 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-369.ged8ee0f2.el9 2024-09-20T19:34:55.165 INFO:teuthology.task.install:The correct ceph version 19.1.1-369.ged8ee0f2 is installed. 2024-09-20T19:34:55.167 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-20T19:34:55.167 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:34:55.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-20T19:34:55.193 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-20T19:34:55.193 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-20T19:34:55.229 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-20T19:34:55.230 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:34:55.230 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-09-20T19:34:55.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-20T19:34:55.347 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-20T19:34:55.347 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/daemon-helper 2024-09-20T19:34:55.372 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-20T19:34:55.437 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-20T19:34:55.437 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:34:55.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-20T19:34:55.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-20T19:34:55.526 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-20T19:34:55.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-20T19:34:55.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-20T19:34:55.615 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-20T19:34:55.615 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:34:55.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-09-20T19:34:55.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-20T19:34:55.703 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-20T19:34:55.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/stdin-killer 2024-09-20T19:34:55.728 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-20T19:34:55.792 INFO:teuthology.run_tasks:Running task print... 2024-09-20T19:34:55.801 INFO:teuthology.task.print:**** done install task... 2024-09-20T19:34:55.801 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-20T19:34:55.927 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'squid', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'global': {'mon pg warn min per osd': 0}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}}, 'image': 'quay.ceph.io/ceph-ci/ceph:squid', 'roleless': True, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': 'e8baca1eac6c3c9c88bf2ea4493e448bd54936a7'} 2024-09-20T19:34:55.928 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:squid 2024-09-20T19:34:55.928 INFO:tasks.cephadm:Cluster fsid is 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:34:55.928 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-20T19:34:55.928 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-09-20T19:34:55.929 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi114': '172.21.15.114', 'mon.smithi160': '172.21.15.160'} 2024-09-20T19:34:55.929 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-20T19:34:55.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo hostname $(hostname -s) 2024-09-20T19:34:55.954 DEBUG:teuthology.orchestra.run.smithi160:> sudo hostname $(hostname -s) 2024-09-20T19:34:55.979 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for squid 2024-09-20T19:34:55.980 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:34:56.120 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e8baca1eac6c3c9c88bf2ea4493e448bd54936a7', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83188/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5127-ge8baca1e', 'node_name': '172.21.2.6+braggi06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5127.ge8baca1e'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-adk-testing-2024-09-20-0910/e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-20 14:53:51.724057', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-adk-testing-2024-09-20-0910', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-adk-testing-2024-09-20-0910/e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-20T19:34:56.346 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 ed8ee0f26ef720e6504cefa8be1a2d0872192de7 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=squid 2024-09-20T19:34:56.348 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/ed8ee0f26ef720e6504cefa8be1a2d0872192de7/centos/9/x86_64/flavors/default/cephadm 2024-09-20T19:34:56.348 INFO:tasks.cephadm:Downloading cephadm from url: https://3.chacra.ceph.com/binaries/ceph/squid/ed8ee0f26ef720e6504cefa8be1a2d0872192de7/centos/9/x86_64/flavors/default/cephadm 2024-09-20T19:34:56.348 DEBUG:teuthology.orchestra.run.smithi114:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/ed8ee0f26ef720e6504cefa8be1a2d0872192de7/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T19:34:56.871 INFO:teuthology.orchestra.run.smithi114.stdout:-rw-r--r--. 1 ubuntu ubuntu 784376 Sep 20 19:34 /home/ubuntu/cephtest/cephadm 2024-09-20T19:34:56.872 DEBUG:teuthology.orchestra.run.smithi160:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/ed8ee0f26ef720e6504cefa8be1a2d0872192de7/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T19:34:57.409 INFO:teuthology.orchestra.run.smithi160.stdout:-rw-r--r--. 1 ubuntu ubuntu 784376 Sep 20 19:34 /home/ubuntu/cephtest/cephadm 2024-09-20T19:34:57.409 DEBUG:teuthology.orchestra.run.smithi114:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T19:34:57.429 DEBUG:teuthology.orchestra.run.smithi160:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T19:34:57.459 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:squid on all hosts... 2024-09-20T19:34:57.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid pull 2024-09-20T19:34:57.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid pull 2024-09-20T19:34:59.097 INFO:teuthology.orchestra.run.smithi160.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:squid... 2024-09-20T19:34:59.109 INFO:teuthology.orchestra.run.smithi114.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:squid... 2024-09-20T19:35:41.536 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-20T19:35:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_version": "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)", 2024-09-20T19:35:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: "image_id": "6acb0a031eab55fe07acc1e5695be48af529a148c2f67dce7aead6591509af30", 2024-09-20T19:35:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "repo_digests": [ 2024-09-20T19:35:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65" 2024-09-20T19:35:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-20T19:35:41.537 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-20T19:35:48.406 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-20T19:35:48.406 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)", 2024-09-20T19:35:48.407 INFO:teuthology.orchestra.run.smithi114.stdout: "image_id": "6acb0a031eab55fe07acc1e5695be48af529a148c2f67dce7aead6591509af30", 2024-09-20T19:35:48.407 INFO:teuthology.orchestra.run.smithi114.stdout: "repo_digests": [ 2024-09-20T19:35:48.407 INFO:teuthology.orchestra.run.smithi114.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65" 2024-09-20T19:35:48.407 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-20T19:35:48.407 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-20T19:35:48.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph 2024-09-20T19:35:48.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph 2024-09-20T19:35:48.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /etc/ceph 2024-09-20T19:35:48.561 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 777 /etc/ceph 2024-09-20T19:35:48.603 INFO:tasks.cephadm:Writing seed config... 2024-09-20T19:35:48.604 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-09-20T19:35:48.604 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-09-20T19:35:48.604 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-09-20T19:35:48.604 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-09-20T19:35:48.604 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-09-20T19:35:48.604 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-09-20T19:35:48.604 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-09-20T19:35:48.604 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-09-20T19:35:48.604 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-09-20T19:35:48.604 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-20T19:35:48.605 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-20T19:35:48.605 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-09-20T19:35:48.605 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-09-20T19:35:48.605 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-09-20T19:35:48.605 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-09-20T19:35:48.605 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-09-20T19:35:48.605 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-09-20T19:35:48.605 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-09-20T19:35:48.605 INFO:tasks.cephadm: override: [mgr] client mount timeout = 30 2024-09-20T19:35:48.605 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-09-20T19:35:48.605 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-20T19:35:48.605 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-20T19:35:48.605 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-20T19:35:48.605 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-20T19:35:48.606 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-20T19:35:48.606 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-20T19:35:48.606 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:35:48.606 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-20T19:35:48.622 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 6a49c7bc-7787-11ef-baf5-efdc52797490 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false client mount timeout = 30 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-20T19:35:48.622 DEBUG:teuthology.orchestra.run.smithi114:mon.smithi114> sudo journalctl -f -n 0 -u ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@mon.smithi114.service 2024-09-20T19:35:48.664 INFO:tasks.cephadm:Bootstrapping... 2024-09-20T19:35:48.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid -v bootstrap --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.114 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:35:48.865 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-20T19:35:48.865 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', '-v', 'bootstrap', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.114', '--skip-admin-label'] 2024-09-20T19:35:48.866 INFO:teuthology.orchestra.run.smithi114.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-20T19:35:48.866 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying podman|docker is present... 2024-09-20T19:35:48.889 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 5.2.2 2024-09-20T19:35:48.889 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying lvm2 is present... 2024-09-20T19:35:48.890 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying time synchronization is in place... 2024-09-20T19:35:48.897 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-20T19:35:48.898 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T19:35:48.909 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-20T19:35:48.910 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-20T19:35:48.910 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-09-20T19:35:48.917 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-09-20T19:35:48.917 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-09-20T19:35:48.918 INFO:teuthology.orchestra.run.smithi114.stdout:Repeating the final host check... 2024-09-20T19:35:48.938 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 5.2.2 2024-09-20T19:35:48.939 INFO:teuthology.orchestra.run.smithi114.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-20T19:35:48.939 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl is present 2024-09-20T19:35:48.939 INFO:teuthology.orchestra.run.smithi114.stdout:lvcreate is present 2024-09-20T19:35:48.945 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-20T19:35:48.945 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T19:35:48.951 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-20T19:35:48.951 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-20T19:35:48.958 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-09-20T19:35:48.965 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-09-20T19:35:48.965 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-09-20T19:35:48.965 INFO:teuthology.orchestra.run.smithi114.stdout:Host looks OK 2024-09-20T19:35:48.965 INFO:teuthology.orchestra.run.smithi114.stdout:Cluster fsid: 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:35:48.965 INFO:teuthology.orchestra.run.smithi114.stdout:Acquiring lock 140005200895280 on /run/cephadm/6a49c7bc-7787-11ef-baf5-efdc52797490.lock 2024-09-20T19:35:48.965 INFO:teuthology.orchestra.run.smithi114.stdout:Lock 140005200895280 acquired on /run/cephadm/6a49c7bc-7787-11ef-baf5-efdc52797490.lock 2024-09-20T19:35:48.965 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying IP 172.21.15.114 port 3300 ... 2024-09-20T19:35:48.966 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying IP 172.21.15.114 port 6789 ... 2024-09-20T19:35:48.966 INFO:teuthology.orchestra.run.smithi114.stdout:Base mon IP(s) is [172.21.15.114:3300, 172.21.15.114:6789], mon addrv is [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-09-20T19:35:48.969 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.114 metric 100 2024-09-20T19:35:48.969 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.114 metric 100 2024-09-20T19:35:48.972 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-20T19:35:48.972 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-20T19:35:48.972 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-20T19:35:48.974 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-20T19:35:48.974 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-20T19:35:48.974 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-20T19:35:48.974 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-20T19:35:48.974 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link noprefixroute 2024-09-20T19:35:48.974 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-20T19:35:48.975 INFO:teuthology.orchestra.run.smithi114.stdout:Mon IP `172.21.15.114` is in CIDR network `172.21.0.0/20` 2024-09-20T19:35:48.975 INFO:teuthology.orchestra.run.smithi114.stdout:Mon IP `172.21.15.114` is in CIDR network `172.21.0.0/20` 2024-09-20T19:35:48.975 INFO:teuthology.orchestra.run.smithi114.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-20T19:35:48.975 INFO:teuthology.orchestra.run.smithi114.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-20T19:35:48.976 INFO:teuthology.orchestra.run.smithi114.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:squid... 2024-09-20T19:35:49.409 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 6acb0a031eab55fe07acc1e5695be48af529a148c2f67dce7aead6591509af30 2024-09-20T19:35:49.409 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:squid... 2024-09-20T19:35:49.409 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Getting image source signatures 2024-09-20T19:35:49.409 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:92ea49144d6cc6e75ae91004beab5c92b7a8b7054f9361617c16e26264c87e92 2024-09-20T19:35:49.409 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-20T19:35:49.409 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying config sha256:6acb0a031eab55fe07acc1e5695be48af529a148c2f67dce7aead6591509af30 2024-09-20T19:35:49.409 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-20T19:35:50.304 INFO:teuthology.orchestra.run.smithi114.stdout:ceph: stdout ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable) 2024-09-20T19:35:50.305 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph version: ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable) 2024-09-20T19:35:50.305 INFO:teuthology.orchestra.run.smithi114.stdout:Extracting ceph user uid/gid from container image... 2024-09-20T19:35:51.335 INFO:teuthology.orchestra.run.smithi114.stdout:stat: stdout 167 167 2024-09-20T19:35:51.336 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial keys... 2024-09-20T19:35:52.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQCXzu1mgJlOJxAAGpzDlbgVD/zkyGjNn/9tOg== 2024-09-20T19:35:53.230 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQCYzu1mUL1/IxAAkMAnOTRJRpt3YtFfcDOpwg== 2024-09-20T19:35:54.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQCZzu1mYVYjIRAAS0dcyiinEqJalnBBX01JJA== 2024-09-20T19:35:54.111 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial monmap... 2024-09-20T19:35:55.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T19:35:55.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-20T19:35:55.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:35:55.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T19:35:55.058 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool for smithi114 [v2:172.21.15.114:3300,v1:172.21.15.114:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T19:35:55.059 INFO:teuthology.orchestra.run.smithi114.stdout:setting min_mon_release = quincy 2024-09-20T19:35:55.059 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: set fsid to 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:35:55.059 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T19:35:55.059 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:35:55.059 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mon... 2024-09-20T19:35:55.998 INFO:teuthology.orchestra.run.smithi114.stdout:create mon.smithi114 on 2024-09-20T19:35:56.212 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-20T19:35:56.370 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-20T19:35:56.546 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-6a49c7bc-7787-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-6a49c7bc-7787-11ef-baf5-efdc52797490.target. 2024-09-20T19:35:56.546 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-6a49c7bc-7787-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-6a49c7bc-7787-11ef-baf5-efdc52797490.target. 2024-09-20T19:35:56.795 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@mon.smithi114 2024-09-20T19:35:56.795 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@mon.smithi114.service: Unit ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@mon.smithi114.service not loaded. 2024-09-20T19:35:56.976 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6a49c7bc-7787-11ef-baf5-efdc52797490.target.wants/ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@mon.smithi114.service → /etc/systemd/system/ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@.service. 2024-09-20T19:35:57.481 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T19:35:57.481 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-20T19:35:57.488 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T19:35:57.488 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-20T19:35:57.488 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-20T19:35:57.488 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-09-20T19:35:57.488 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon to start... 2024-09-20T19:35:57.489 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon... 2024-09-20T19:35:57.566 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:35:57 smithi114 systemd[1]: Started Ceph mon.smithi114 for 6a49c7bc-7787-11ef-baf5-efdc52797490. 2024-09-20T19:35:58.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout cluster: 2024-09-20T19:35:58.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout id: 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:35:58.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-20T19:35:58.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-20T19:35:58.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout services: 2024-09-20T19:35:58.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi114 (age 0.513982s) 2024-09-20T19:35:58.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-20T19:35:58.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-20T19:35:58.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-20T19:35:58.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout data: 2024-09-20T19:35:58.478 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-20T19:35:58.479 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-20T19:35:58.479 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-20T19:35:58.479 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pgs: 2024-09-20T19:35:58.479 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-20T19:35:58.479 INFO:teuthology.orchestra.run.smithi114.stdout:mon is available 2024-09-20T19:35:58.479 INFO:teuthology.orchestra.run.smithi114.stdout:Assimilating anything we can from ceph.conf... 2024-09-20T19:35:58.600 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:35:58 smithi114 ceph-mon[27896]: mon.smithi114 is new leader, mons smithi114 in quorum (ranks 0) 2024-09-20T19:35:58.600 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:35:58 smithi114 ceph-mon[27896]: monmap epoch 1 2024-09-20T19:35:58.601 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:35:58 smithi114 ceph-mon[27896]: fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:35:58.601 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:35:58 smithi114 ceph-mon[27896]: last_changed 2024-09-20T19:35:54.449562+0000 2024-09-20T19:35:58.601 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:35:58 smithi114 ceph-mon[27896]: created 2024-09-20T19:35:54.449562+0000 2024-09-20T19:35:58.601 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:35:58 smithi114 ceph-mon[27896]: min_mon_release 19 (squid) 2024-09-20T19:35:58.601 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:35:58 smithi114 ceph-mon[27896]: election_strategy: 1 2024-09-20T19:35:58.601 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:35:58 smithi114 ceph-mon[27896]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.smithi114 2024-09-20T19:35:58.601 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:35:58 smithi114 ceph-mon[27896]: fsmap 2024-09-20T19:35:58.601 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:35:58 smithi114 ceph-mon[27896]: osdmap e1: 0 total, 0 up, 0 in 2024-09-20T19:35:58.602 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:35:58 smithi114 ceph-mon[27896]: mgrmap e1: no daemons active 2024-09-20T19:35:58.602 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:35:58 smithi114 ceph-mon[27896]: from='client.? 172.21.15.114:0/847395332' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-20T19:36:01.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:01 smithi114 ceph-mon[27896]: from='client.? 172.21.15.114:0/1722668118' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-20T19:36:01.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:01 smithi114 ceph-mon[27896]: from='client.? 172.21.15.114:0/1722668118' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-20T19:36:08.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-20T19:36:08.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-09-20T19:36:08.074 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:36:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-09-20T19:36:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-20T19:36:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-20T19:36:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-20T19:36:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-20T19:36:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-20T19:36:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-20T19:36:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-20T19:36:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-20T19:36:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-09-20T19:36:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-20T19:36:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-20T19:36:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-20T19:36:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-20T19:36:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-20T19:36:08.076 INFO:teuthology.orchestra.run.smithi114.stdout:Generating new minimal ceph.conf... 2024-09-20T19:36:08.714 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:08 smithi114 ceph-mon[27896]: from='client.? 172.21.15.114:0/2239113071' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:36:09.193 INFO:teuthology.orchestra.run.smithi114.stdout:Restarting the monitor... 2024-09-20T19:36:09.444 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:09 smithi114 systemd[1]: Stopping Ceph mon.smithi114 for 6a49c7bc-7787-11ef-baf5-efdc52797490... 2024-09-20T19:36:09.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[27892]: 2024-09-20T19:36:09.442+0000 7fd3f841e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi114 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T19:36:09.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[27892]: 2024-09-20T19:36:09.442+0000 7fd3f841e640 -1 mon.smithi114@0(leader) e1 *** Got Signal Terminated *** 2024-09-20T19:36:09.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:09 smithi114 podman[28138]: 2024-09-20 19:36:09.514707197 +0000 UTC m=+0.163957534 container died ef54a2d37e59dc9b5afd7a251dcb8ac1905faaa9dbd056805ca4a0c453c4e705 (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-ed8ee0f, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T19:36:10.183 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:09 smithi114 podman[28138]: 2024-09-20 19:36:09.817520406 +0000 UTC m=+0.466770738 container cleanup ef54a2d37e59dc9b5afd7a251dcb8ac1905faaa9dbd056805ca4a0c453c4e705 (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-ed8ee0f) 2024-09-20T19:36:10.183 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:09 smithi114 bash[28138]: ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114 2024-09-20T19:36:10.183 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:09 smithi114 podman[28151]: 2024-09-20 19:36:09.976889263 +0000 UTC m=+0.459806067 container remove ef54a2d37e59dc9b5afd7a251dcb8ac1905faaa9dbd056805ca4a0c453c4e705 (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114, RELEASE=squid-ed8ee0f, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T19:36:10.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 systemd[1]: ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@mon.smithi114.service: Deactivated successfully. 2024-09-20T19:36:10.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 systemd[1]: Stopped Ceph mon.smithi114 for 6a49c7bc-7787-11ef-baf5-efdc52797490. 2024-09-20T19:36:10.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 systemd[1]: Starting Ceph mon.smithi114 for 6a49c7bc-7787-11ef-baf5-efdc52797490... 2024-09-20T19:36:10.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 podman[28241]: 2024-09-20 19:36:10.312581755 +0000 UTC m=+0.021306600 image pull 6acb0a031eab55fe07acc1e5695be48af529a148c2f67dce7aead6591509af30 quay.ceph.io/ceph-ci/ceph:squid 2024-09-20T19:36:10.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 podman[28241]: 2024-09-20 19:36:10.419350918 +0000 UTC m=+0.128075752 container create ed7f013f251354937f820546027918cc28565eae298d1e9671d882c09e8fcd7f (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-ed8ee0f, org.label-schema.build-date=20240731) 2024-09-20T19:36:10.743 INFO:teuthology.orchestra.run.smithi114.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-20T19:36:10.848 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 podman[28241]: 2024-09-20 19:36:10.654941506 +0000 UTC m=+0.363666355 container init ed7f013f251354937f820546027918cc28565eae298d1e9671d882c09e8fcd7f (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=squid-ed8ee0f, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-20T19:36:10.849 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 podman[28241]: 2024-09-20 19:36:10.657870483 +0000 UTC m=+0.366595317 container start ed7f013f251354937f820546027918cc28565eae298d1e9671d882c09e8fcd7f (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=squid-ed8ee0f, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T19:36:10.849 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T19:36:10.849 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable), process ceph-mon, pid 2 2024-09-20T19:36:10.849 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: pidfile_write: ignore empty --pid-file 2024-09-20T19:36:10.849 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: load: jerasure load: lrc 2024-09-20T19:36:10.849 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: RocksDB version: 7.9.2 2024-09-20T19:36:10.849 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Git sha 0 2024-09-20T19:36:10.849 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Compile date 2024-09-19 20:45:27 2024-09-20T19:36:10.849 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: DB SUMMARY 2024-09-20T19:36:10.849 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: DB Session ID: 3408L1WFS72J6FHNEVE8 2024-09-20T19:36:10.850 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: CURRENT file: CURRENT 2024-09-20T19:36:10.850 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T19:36:10.850 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-20T19:36:10.850 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi114/store.db dir, Total Num: 1, files: 000008.sst 2024-09-20T19:36:10.850 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi114/store.db: 000009.log size: 95861 ; 2024-09-20T19:36:10.850 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.error_if_exists: 0 2024-09-20T19:36:10.850 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.create_if_missing: 0 2024-09-20T19:36:10.850 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.paranoid_checks: 1 2024-09-20T19:36:10.850 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T19:36:10.850 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T19:36:10.850 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T19:36:10.850 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.env: 0x55efb5171940 2024-09-20T19:36:10.851 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T19:36:10.851 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.info_log: 0x55efb61a38c0 2024-09-20T19:36:10.851 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T19:36:10.851 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.statistics: (nil) 2024-09-20T19:36:10.851 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.use_fsync: 0 2024-09-20T19:36:10.852 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_log_file_size: 0 2024-09-20T19:36:10.852 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T19:36:10.853 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T19:36:10.853 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T19:36:10.853 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T19:36:10.853 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.allow_fallocate: 1 2024-09-20T19:36:10.853 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T19:36:10.853 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T19:36:10.853 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.use_direct_reads: 0 2024-09-20T19:36:10.853 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T19:36:10.853 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T19:36:10.853 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.db_log_dir: 2024-09-20T19:36:10.853 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.wal_dir: 2024-09-20T19:36:10.854 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T19:36:10.854 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T19:36:10.854 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T19:36:10.854 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T19:36:10.854 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T19:36:10.854 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T19:36:10.854 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T19:36:10.854 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T19:36:10.854 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.write_buffer_manager: 0x55efb61b2dc0 2024-09-20T19:36:10.854 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T19:36:10.854 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T19:36:10.855 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T19:36:10.855 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T19:36:10.855 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T19:36:10.855 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T19:36:10.855 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T19:36:10.855 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T19:36:10.855 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.unordered_write: 0 2024-09-20T19:36:10.855 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T19:36:10.855 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T19:36:10.855 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T19:36:10.856 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T19:36:10.856 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.row_cache: None 2024-09-20T19:36:10.856 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.wal_filter: None 2024-09-20T19:36:10.856 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T19:36:10.856 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T19:36:10.856 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.two_write_queues: 0 2024-09-20T19:36:10.856 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T19:36:10.856 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.wal_compression: 0 2024-09-20T19:36:10.856 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.atomic_flush: 0 2024-09-20T19:36:10.856 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T19:36:10.856 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T19:36:10.856 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T19:36:10.857 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.log_readahead_size: 0 2024-09-20T19:36:10.857 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T19:36:10.857 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T19:36:10.857 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T19:36:10.857 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T19:36:10.857 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T19:36:10.857 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T19:36:10.857 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T19:36:10.857 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_background_jobs: 2 2024-09-20T19:36:10.857 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_background_compactions: -1 2024-09-20T19:36:10.857 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_subcompactions: 1 2024-09-20T19:36:10.857 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T19:36:10.858 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T19:36:10.858 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T19:36:10.858 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T19:36:10.858 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T19:36:10.858 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T19:36:10.858 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T19:36:10.858 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T19:36:10.858 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_open_files: -1 2024-09-20T19:36:10.858 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T19:36:10.858 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T19:36:10.858 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T19:36:10.858 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T19:36:10.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_background_flushes: -1 2024-09-20T19:36:10.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Compression algorithms supported: 2024-09-20T19:36:10.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: kZSTD supported: 0 2024-09-20T19:36:10.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: kXpressCompression supported: 0 2024-09-20T19:36:10.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: kBZip2Compression supported: 0 2024-09-20T19:36:10.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T19:36:10.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: kLZ4Compression supported: 1 2024-09-20T19:36:10.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: kZlibCompression supported: 1 2024-09-20T19:36:10.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T19:36:10.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: kSnappyCompression supported: 1 2024-09-20T19:36:10.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T19:36:10.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T19:36:10.860 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi114/store.db/MANIFEST-000010 2024-09-20T19:36:10.860 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T19:36:10.860 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T19:36:10.860 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.merge_operator: 2024-09-20T19:36:10.860 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compaction_filter: None 2024-09-20T19:36:10.860 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T19:36:10.860 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T19:36:10.860 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T19:36:10.860 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T19:36:10.860 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55efb61a3520) 2024-09-20T19:36:10.860 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-09-20T19:36:10.860 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T19:36:10.861 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T19:36:10.861 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-09-20T19:36:10.861 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: index_type: 0 2024-09-20T19:36:10.861 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: data_block_index_type: 0 2024-09-20T19:36:10.861 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: index_shortening: 1 2024-09-20T19:36:10.861 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T19:36:10.861 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: checksum: 4 2024-09-20T19:36:10.862 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: no_block_cache: 0 2024-09-20T19:36:10.862 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_cache: 0x55efb61971f0 2024-09-20T19:36:10.862 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-09-20T19:36:10.862 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_cache_options: 2024-09-20T19:36:10.862 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: capacity : 536870912 2024-09-20T19:36:10.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: num_shard_bits : 4 2024-09-20T19:36:10.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: strict_capacity_limit : 0 2024-09-20T19:36:10.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-09-20T19:36:10.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_cache_compressed: (nil) 2024-09-20T19:36:10.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: persistent_cache: (nil) 2024-09-20T19:36:10.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_size: 4096 2024-09-20T19:36:10.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_size_deviation: 10 2024-09-20T19:36:10.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_restart_interval: 16 2024-09-20T19:36:10.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: index_block_restart_interval: 1 2024-09-20T19:36:10.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: metadata_block_size: 4096 2024-09-20T19:36:10.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: partition_filters: 0 2024-09-20T19:36:10.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: use_delta_encoding: 1 2024-09-20T19:36:10.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: filter_policy: bloomfilter 2024-09-20T19:36:10.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: whole_key_filtering: 1 2024-09-20T19:36:10.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: verify_compression: 0 2024-09-20T19:36:10.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-09-20T19:36:10.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: format_version: 5 2024-09-20T19:36:10.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: enable_index_compression: 1 2024-09-20T19:36:10.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_align: 0 2024-09-20T19:36:10.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: max_auto_readahead_size: 262144 2024-09-20T19:36:10.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: prepopulate_block_cache: 0 2024-09-20T19:36:10.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: initial_auto_readahead_size: 8192 2024-09-20T19:36:10.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T19:36:10.865 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T19:36:10.865 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T19:36:10.865 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compression: NoCompression 2024-09-20T19:36:10.865 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T19:36:10.865 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T19:36:10.865 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T19:36:10.865 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.num_levels: 7 2024-09-20T19:36:10.865 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T19:36:10.865 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T19:36:10.865 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T19:36:10.865 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T19:36:10.865 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T19:36:10.865 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T19:36:10.866 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T19:36:10.866 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:36:10.866 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T19:36:10.866 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T19:36:10.866 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:36:10.866 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:36:10.866 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T19:36:10.866 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T19:36:10.866 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T19:36:10.866 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T19:36:10.866 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T19:36:10.866 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T19:36:10.866 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T19:36:10.866 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T19:36:10.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T19:36:10.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T19:36:10.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T19:36:10.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T19:36:10.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T19:36:10.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T19:36:10.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T19:36:10.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T19:36:10.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T19:36:10.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T19:36:10.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T19:36:10.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T19:36:10.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T19:36:10.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T19:36:10.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T19:36:10.868 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T19:36:10.868 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T19:36:10.868 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T19:36:10.868 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T19:36:10.868 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T19:36:10.868 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T19:36:10.868 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T19:36:10.868 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T19:36:10.868 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T19:36:10.869 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T19:36:10.869 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T19:36:10.869 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T19:36:10.869 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T19:36:10.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T19:36:10.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T19:36:10.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T19:36:10.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T19:36:10.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T19:36:10.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T19:36:10.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.inplace_update_support: 0 2024-09-20T19:36:10.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T19:36:10.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T19:36:10.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T19:36:10.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T19:36:10.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.bloom_locality: 0 2024-09-20T19:36:10.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.max_successive_merges: 0 2024-09-20T19:36:10.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T19:36:10.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T19:36:10.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T19:36:10.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T19:36:10.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.ttl: 2592000 2024-09-20T19:36:10.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T19:36:10.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T19:36:10.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T19:36:10.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.enable_blob_files: false 2024-09-20T19:36:10.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.min_blob_size: 0 2024-09-20T19:36:10.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T19:36:10.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T19:36:10.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T19:36:10.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T19:36:10.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T19:36:10.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T19:36:10.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T19:36:10.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T19:36:10.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi114/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-20T19:36:10.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-20T19:36:10.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 06abd419-e914-43fb-9ab5-7ccfaf8462f5 2024-09-20T19:36:10.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726860970683831, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-20T19:36:10.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-20T19:36:10.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726860970684702, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 91862, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 288, "table_properties": {"data_size": 89903, "index_size": 279, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11912, "raw_average_key_size": 48, "raw_value_size": 83271, "raw_average_value_size": 337, "num_data_blocks": 12, "num_entries": 247, "num_filter_entries": 247, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726860970, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "06abd419-e914-43fb-9ab5-7ccfaf8462f5", "db_session_id": "3408L1WFS72J6FHNEVE8", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-20T19:36:10.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726860970684798, "job": 1, "event": "recovery_finished"} 2024-09-20T19:36:10.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-20T19:36:10.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi114/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T19:36:10.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55efb61c4e00 2024-09-20T19:36:10.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: DB pointer 0x55efb62c0000 2024-09-20T19:36:10.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T19:36:10.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T19:36:10.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ** DB Stats ** 2024-09-20T19:36:10.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:36:10.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T19:36:10.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:36:10.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:36:10.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T19:36:10.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T19:36:10.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T19:36:10.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-09-20T19:36:10.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-20T19:36:10.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:36:10.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T19:36:10.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: L0 2/0 91.61 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:36:10.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Sum 2/0 91.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:36:10.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:36:10.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-09-20T19:36:10.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-20T19:36:10.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T19:36:10.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T19:36:10.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T19:36:10.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-09-20T19:36:10.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T19:36:10.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-09-20T19:36:10.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T19:36:10.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T19:36:10.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T19:36:10.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T19:36:10.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T19:36:10.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T19:36:10.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Cumulative compaction: 0.00 GB write, 29.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:36:10.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Interval compaction: 0.00 GB write, 29.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T19:36:10.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T19:36:10.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Block cache BinnedLRUCache@0x55efb61971f0#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-20T19:36:10.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-20T19:36:10.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-09-20T19:36:10.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T19:36:10.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: starting mon.smithi114 rank 0 at public addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] at bind addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi114 fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:36:10.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: mon.smithi114@-1(???) e1 preinit fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:36:10.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: mon.smithi114@-1(???).mds e1 new map 2024-09-20T19:36:10.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: mon.smithi114@-1(???).mds e1 print_map 2024-09-20T19:36:10.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: e1 2024-09-20T19:36:10.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: btime 2024-09-20T19:35:57:433642+0000 2024-09-20T19:36:10.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T19:36:10.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-20T19:36:10.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: legacy client fscid: -1 2024-09-20T19:36:10.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-09-20T19:36:10.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: No filesystems configured 2024-09-20T19:36:10.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: mon.smithi114@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T19:36:10.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: mon.smithi114@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:36:10.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: mon.smithi114@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:36:10.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: mon.smithi114@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T19:36:10.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: mon.smithi114@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-20T19:36:10.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 bash[28241]: ed7f013f251354937f820546027918cc28565eae298d1e9671d882c09e8fcd7f 2024-09-20T19:36:10.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 systemd[1]: Started Ceph mon.smithi114 for 6a49c7bc-7787-11ef-baf5-efdc52797490. 2024-09-20T19:36:10.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: mon.smithi114 is new leader, mons smithi114 in quorum (ranks 0) 2024-09-20T19:36:10.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: monmap epoch 1 2024-09-20T19:36:10.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:36:10.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: last_changed 2024-09-20T19:35:54.449562+0000 2024-09-20T19:36:10.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: created 2024-09-20T19:35:54.449562+0000 2024-09-20T19:36:10.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: min_mon_release 19 (squid) 2024-09-20T19:36:10.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: election_strategy: 1 2024-09-20T19:36:10.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.smithi114 2024-09-20T19:36:10.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: fsmap 2024-09-20T19:36:10.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: osdmap e1: 0 total, 0 up, 0 in 2024-09-20T19:36:10.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:10 smithi114 ceph-mon[28260]: mgrmap e1: no daemons active 2024-09-20T19:36:11.807 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-20T19:36:11.828 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:36:11.828 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mgr... 2024-09-20T19:36:11.830 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-20T19:36:11.830 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-20T19:36:11.830 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:8443 ... 2024-09-20T19:36:12.074 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@mgr.smithi114.rvhgji 2024-09-20T19:36:12.074 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@mgr.smithi114.rvhgji.service: Unit ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@mgr.smithi114.rvhgji.service not loaded. 2024-09-20T19:36:12.219 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6a49c7bc-7787-11ef-baf5-efdc52797490.target.wants/ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@mgr.smithi114.rvhgji.service → /etc/systemd/system/ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@.service. 2024-09-20T19:36:12.456 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:12 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/406597775' entity='client.admin' 2024-09-20T19:36:12.806 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T19:36:12.806 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-20T19:36:12.813 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T19:36:12.813 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-20T19:36:12.813 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-20T19:36:12.813 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-09-20T19:36:12.824 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T19:36:12.824 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-20T19:36:12.833 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T19:36:12.833 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-20T19:36:12.833 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-20T19:36:12.833 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-09-20T19:36:12.833 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr to start... 2024-09-20T19:36:12.833 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr... 2024-09-20T19:36:13.714 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:13 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3508190315' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T19:36:16.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-20T19:36:16.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:36:16.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "6a49c7bc-7787-11ef-baf5-efdc52797490", 2024-09-20T19:36:16.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T19:36:16.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T19:36:16.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T19:36:16.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T19:36:16.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "smithi114" 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:36:16.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T19:36:16.905 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T19:36:16.905 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T19:36:16.905 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T19:36:16.908 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T19:36:16.908 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T19:36:16.908 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:36:16.908 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T19:36:16.908 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T19:36:16.909 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T19:36:16.909 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T19:36:16.909 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T19:36:16.909 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T19:36:16.909 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T19:36:16.909 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T19:36:16.909 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T19:36:16.909 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:36:16.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T19:36:16.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:36:16.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "btime": "2024-09-20T19:35:57:433642+0000", 2024-09-20T19:36:16.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T19:36:16.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T19:36:16.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:36:16.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T19:36:16.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-20T19:36:16.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T19:36:16.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T19:36:16.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T19:36:16.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T19:36:16.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T19:36:16.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:36:16.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:36:16.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:36:16.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T19:36:16.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:36:16.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T19:35:57.434308+0000", 2024-09-20T19:36:16.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:36:16.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:36:16.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T19:36:16.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:36:16.911 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (1/15)... 2024-09-20T19:36:17.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:17 smithi114 ceph-mon[28260]: Activating manager daemon smithi114.rvhgji 2024-09-20T19:36:17.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:17 smithi114 ceph-mon[28260]: mgrmap e2: smithi114.rvhgji(active, starting, since 0.00293298s) 2024-09-20T19:36:17.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:17 smithi114 ceph-mon[28260]: from='mgr.14100 172.21.15.114:0/2014185129' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:36:17.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:17 smithi114 ceph-mon[28260]: from='mgr.14100 172.21.15.114:0/2014185129' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:36:17.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:17 smithi114 ceph-mon[28260]: from='mgr.14100 172.21.15.114:0/2014185129' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:36:17.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:17 smithi114 ceph-mon[28260]: from='mgr.14100 172.21.15.114:0/2014185129' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-09-20T19:36:17.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:17 smithi114 ceph-mon[28260]: from='mgr.14100 172.21.15.114:0/2014185129' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mgr metadata", "who": "smithi114.rvhgji", "id": "smithi114.rvhgji"}]: dispatch 2024-09-20T19:36:17.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:17 smithi114 ceph-mon[28260]: Manager daemon smithi114.rvhgji is now available 2024-09-20T19:36:17.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:17 smithi114 ceph-mon[28260]: from='mgr.14100 172.21.15.114:0/2014185129' entity='mgr.smithi114.rvhgji' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.rvhgji/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:36:17.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:17 smithi114 ceph-mon[28260]: from='mgr.14100 172.21.15.114:0/2014185129' entity='mgr.smithi114.rvhgji' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.rvhgji/trash_purge_schedule"}]: dispatch 2024-09-20T19:36:17.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:17 smithi114 ceph-mon[28260]: from='mgr.14100 172.21.15.114:0/2014185129' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:17.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:17 smithi114 ceph-mon[28260]: from='mgr.14100 172.21.15.114:0/2014185129' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:17.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:17 smithi114 ceph-mon[28260]: from='mgr.14100 172.21.15.114:0/2014185129' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:19.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:19 smithi114 ceph-mon[28260]: mgrmap e3: smithi114.rvhgji(active, since 1.00614s) 2024-09-20T19:36:20.189 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-20T19:36:20.189 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:36:20.189 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "6a49c7bc-7787-11ef-baf5-efdc52797490", 2024-09-20T19:36:20.189 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T19:36:20.189 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T19:36:20.189 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T19:36:20.189 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T19:36:20.190 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:36:20.190 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T19:36:20.190 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T19:36:20.190 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-09-20T19:36:20.190 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:36:20.190 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T19:36:20.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "smithi114" 2024-09-20T19:36:20.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:36:20.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-20T19:36:20.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T19:36:20.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:36:20.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-20T19:36:20.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T19:36:20.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:36:20.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T19:36:20.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:36:20.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T19:36:20.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T19:36:20.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T19:36:20.191 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T19:36:20.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T19:36:20.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T19:36:20.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:36:20.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T19:36:20.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T19:36:20.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T19:36:20.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T19:36:20.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T19:36:20.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T19:36:20.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T19:36:20.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T19:36:20.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T19:36:20.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:36:20.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T19:36:20.192 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:36:20.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "btime": "2024-09-20T19:35:57:433642+0000", 2024-09-20T19:36:20.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T19:36:20.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T19:36:20.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:36:20.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T19:36:20.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T19:36:20.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T19:36:20.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T19:36:20.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T19:36:20.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T19:36:20.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T19:36:20.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-20T19:36:20.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:36:20.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:36:20.193 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T19:36:20.194 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T19:36:20.194 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T19:35:57.434308+0000", 2024-09-20T19:36:20.194 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T19:36:20.194 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-20T19:36:20.194 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T19:36:20.194 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:36:20.194 INFO:teuthology.orchestra.run.smithi114.stdout:mgr is available 2024-09-20T19:36:20.714 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:20 smithi114 ceph-mon[28260]: mgrmap e4: smithi114.rvhgji(active, since 2s) 2024-09-20T19:36:20.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:20 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3559487753' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T19:36:21.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-20T19:36:21.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-09-20T19:36:21.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:36:21.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-09-20T19:36:21.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-20T19:36:21.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-20T19:36:21.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-20T19:36:21.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-20T19:36:21.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-20T19:36:21.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-20T19:36:21.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-20T19:36:21.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-20T19:36:21.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-09-20T19:36:21.267 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-20T19:36:21.267 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-20T19:36:21.267 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-20T19:36:21.267 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-20T19:36:21.267 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-20T19:36:21.267 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling cephadm module... 2024-09-20T19:36:21.689 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:21 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3030238969' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-20T19:36:22.851 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:22 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1959351539' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-20T19:36:23.714 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:23 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1959351539' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-20T19:36:23.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:23 smithi114 ceph-mon[28260]: mgrmap e5: smithi114.rvhgji(active, since 5s) 2024-09-20T19:36:24.286 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:36:24.286 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-20T19:36:24.286 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T19:36:24.286 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "smithi114.rvhgji", 2024-09-20T19:36:24.287 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-20T19:36:24.287 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:36:24.287 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-09-20T19:36:24.287 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 5... 2024-09-20T19:36:24.702 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:24 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/2455274086' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-20T19:36:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: Active manager daemon smithi114.rvhgji restarted 2024-09-20T19:36:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: Activating manager daemon smithi114.rvhgji 2024-09-20T19:36:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: osdmap e2: 0 total, 0 up, 0 in 2024-09-20T19:36:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: mgrmap e6: smithi114.rvhgji(active, starting, since 0.00287822s) 2024-09-20T19:36:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-09-20T19:36:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mgr metadata", "who": "smithi114.rvhgji", "id": "smithi114.rvhgji"}]: dispatch 2024-09-20T19:36:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:36:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:36:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:36:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: Manager daemon smithi114.rvhgji is now available 2024-09-20T19:36:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:27.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:36:27.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:36:27.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.rvhgji/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:36:27.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:27 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.rvhgji/trash_purge_schedule"}]: dispatch 2024-09-20T19:36:28.672 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:28 smithi114 ceph-mon[28260]: Found migration_current of "None". Setting to last migration. 2024-09-20T19:36:28.672 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:28 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:28.672 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:28 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:28.672 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:28 smithi114 ceph-mon[28260]: mgrmap e7: smithi114.rvhgji(active, since 1.0052s) 2024-09-20T19:36:28.926 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:36:28.926 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-20T19:36:28.926 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-20T19:36:28.926 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:36:28.926 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 5 is available 2024-09-20T19:36:28.926 INFO:teuthology.orchestra.run.smithi114.stdout:Setting orchestrator backend to cephadm... 2024-09-20T19:36:29.524 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:29 smithi114 ceph-mon[28260]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-20T19:36:29.524 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:29 smithi114 ceph-mon[28260]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-20T19:36:29.524 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:29 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:36:30.661 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:30 smithi114 ceph-mon[28260]: [20/Sep/2024:19:36:29] ENGINE Bus STARTING 2024-09-20T19:36:30.661 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:30 smithi114 ceph-mon[28260]: [20/Sep/2024:19:36:29] ENGINE Serving on http://172.21.15.114:8765 2024-09-20T19:36:30.661 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:30 smithi114 ceph-mon[28260]: [20/Sep/2024:19:36:29] ENGINE Serving on https://172.21.15.114:7150 2024-09-20T19:36:30.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:30 smithi114 ceph-mon[28260]: [20/Sep/2024:19:36:29] ENGINE Bus STARTED 2024-09-20T19:36:30.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:30 smithi114 ceph-mon[28260]: [20/Sep/2024:19:36:29] ENGINE Error in HTTPServer.serve 2024-09-20T19:36:30.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Traceback (most recent call last): 2024-09-20T19:36:30.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:36:30.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:36:30.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:36:30.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: self._run(expiration_interval) 2024-09-20T19:36:30.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:36:30.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:36:30.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:36:30.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:36:30.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:36:30.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: s = self.context.wrap_socket( 2024-09-20T19:36:30.663 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:36:30.663 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: return self.sslsocket_class._create( 2024-09-20T19:36:30.663 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:36:30.663 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: self.do_handshake() 2024-09-20T19:36:30.663 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:36:30.663 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: self._sslobj.do_handshake() 2024-09-20T19:36:30.663 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:36:30.663 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:30 smithi114 ceph-mon[28260]: mgrmap e8: smithi114.rvhgji(active, since 2s) 2024-09-20T19:36:30.663 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:30 smithi114 ceph-mon[28260]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:36:30.663 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:30 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:30.663 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:30 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:36:31.221 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-20T19:36:31.222 INFO:teuthology.orchestra.run.smithi114.stdout:Generating ssh key... 2024-09-20T19:36:31.912 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:31 smithi114 ceph-mon[28260]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:36:32.922 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:32 smithi114 ceph-mon[28260]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:36:32.922 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:32 smithi114 ceph-mon[28260]: Generating ssh key... 2024-09-20T19:36:32.922 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:32 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:32.923 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:32 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:33.476 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC83OTk+RSChAQKaDsLmkRr7EmZt7IHbiPRbgHW8gJWdT+W7ZmRhXQwws2B0M9tAbX5wxq48W/o1O9BhxbTUZG1Wgav4pbi4qPNdIfhcabDP496RUUzXkvTkBEEQeLcOcYAIfevCwqiIndeNXHxdRM3uQNPHd3P4CHG8IxCb9VE6edFMKZQ1UGATBEl5y5rryuRNjWESw1U2Cqi64VaKkjBAsFFgxLT8m0+uvY2SPpEW31lPYDmp3zEuPBW/f7Wf6MLYojNER43kLvpM8oNG6IVGL4rl867634B9lc5KrQ9cOp5sVfW5TCl6pGFUZ1awo8dMfSzIOFImaKBtiDupfe08xy5av/GSp+sWTUE/vJBlftgobdut5RfAd5Ab1LpAGikYNInfBNgosvHy1itAfRoCgo1boHc5FQfavkbwJpeG1xpR1k98i7/euhgX4LX947KZMYmWXJn11NiOC/TKgrsxBBFCh0T/I63dxtqmXqBPRAtssoFOZIv/2Wzvum1xfs= ceph-6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:36:33.476 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-20T19:36:33.476 INFO:teuthology.orchestra.run.smithi114.stdout:Adding key to root@localhost authorized_keys... 2024-09-20T19:36:33.478 INFO:teuthology.orchestra.run.smithi114.stdout:Adding host smithi114... 2024-09-20T19:36:34.160 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:33 smithi114 ceph-mon[28260]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:36:35.091 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:34 smithi114 ceph-mon[28260]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "addr": "172.21.15.114", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:36:36.214 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:35 smithi114 ceph-mon[28260]: Deploying cephadm binary to smithi114 2024-09-20T19:36:38.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Added host 'smithi114' with addr '172.21.15.114' 2024-09-20T19:36:38.034 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying mon service with default placement... 2024-09-20T19:36:38.343 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:38 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:38.343 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:38 smithi114 ceph-mon[28260]: Added host smithi114 2024-09-20T19:36:38.343 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:38 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:36:39.847 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-20T19:36:39.847 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying mgr service with default placement... 2024-09-20T19:36:39.964 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:39 smithi114 ceph-mon[28260]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:36:39.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:39 smithi114 ceph-mon[28260]: Saving service mon spec with placement count:5 2024-09-20T19:36:39.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:39 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:41.283 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-20T19:36:41.283 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying crash service with default placement... 2024-09-20T19:36:41.590 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:41 smithi114 ceph-mon[28260]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:36:41.590 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:41 smithi114 ceph-mon[28260]: Saving service mgr spec with placement count:2 2024-09-20T19:36:41.590 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:41 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:41.590 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:41 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:41.590 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:41 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:43.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-09-20T19:36:43.000 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying ceph-exporter service with default placement... 2024-09-20T19:36:43.016 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:42 smithi114 ceph-mon[28260]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:36:43.016 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:42 smithi114 ceph-mon[28260]: Saving service crash spec with placement * 2024-09-20T19:36:43.016 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:42 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:43.017 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:42 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:44.225 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-09-20T19:36:44.225 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying prometheus service with default placement... 2024-09-20T19:36:44.812 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:44 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:44.812 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:44 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:45.714 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:45 smithi114 ceph-mon[28260]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:36:45.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:45 smithi114 ceph-mon[28260]: Saving service ceph-exporter spec with placement * 2024-09-20T19:36:45.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:45 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:45.852 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-09-20T19:36:45.853 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying grafana service with default placement... 2024-09-20T19:36:46.714 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:46 smithi114 ceph-mon[28260]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:36:46.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:46 smithi114 ceph-mon[28260]: Saving service prometheus spec with placement count:1 2024-09-20T19:36:46.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:46 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:47.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-09-20T19:36:47.294 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying node-exporter service with default placement... 2024-09-20T19:36:47.714 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:47 smithi114 ceph-mon[28260]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:36:47.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:47 smithi114 ceph-mon[28260]: Saving service grafana spec with placement count:1 2024-09-20T19:36:48.400 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-09-20T19:36:48.400 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying alertmanager service with default placement... 2024-09-20T19:36:48.758 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:48 smithi114 ceph-mon[28260]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:36:48.758 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:48 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:49.573 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-09-20T19:36:49.822 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:49 smithi114 ceph-mon[28260]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:36:49.822 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:49 smithi114 ceph-mon[28260]: Saving service node-exporter spec with placement * 2024-09-20T19:36:49.822 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:49 smithi114 ceph-mon[28260]: from='mgr.14116 172.21.15.114:0/1989424983' entity='mgr.smithi114.rvhgji' 2024-09-20T19:36:50.719 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:50 smithi114 ceph-mon[28260]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:36:50.719 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:50 smithi114 ceph-mon[28260]: Saving service alertmanager spec with placement count:1 2024-09-20T19:36:50.719 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:50 smithi114 ceph-mon[28260]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:36:50.719 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:50 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/2991340206' entity='client.admin' 2024-09-20T19:36:51.903 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling the dashboard module... 2024-09-20T19:36:52.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:52 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/493938725' entity='client.admin' 2024-09-20T19:36:52.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:52 smithi114 ceph-mon[28260]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:36:53.677 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:53 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1424751349' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-20T19:36:54.579 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:54 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1424751349' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-20T19:36:54.579 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:54 smithi114 ceph-mon[28260]: mgrmap e9: smithi114.rvhgji(active, since 25s) 2024-09-20T19:36:55.199 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:36:55.200 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-20T19:36:55.200 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T19:36:55.200 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "smithi114.rvhgji", 2024-09-20T19:36:55.200 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-20T19:36:55.200 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:36:55.200 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-09-20T19:36:55.200 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 9... 2024-09-20T19:36:55.331 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:55 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3534292329' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-20T19:36:58.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:58 smithi114 ceph-mon[28260]: Active manager daemon smithi114.rvhgji restarted 2024-09-20T19:36:58.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:58 smithi114 ceph-mon[28260]: Activating manager daemon smithi114.rvhgji 2024-09-20T19:36:58.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:58 smithi114 ceph-mon[28260]: osdmap e3: 0 total, 0 up, 0 in 2024-09-20T19:36:58.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:58 smithi114 ceph-mon[28260]: mgrmap e10: smithi114.rvhgji(active, starting, since 0.00340049s) 2024-09-20T19:36:58.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:58 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-09-20T19:36:58.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:58 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mgr metadata", "who": "smithi114.rvhgji", "id": "smithi114.rvhgji"}]: dispatch 2024-09-20T19:36:58.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:58 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:36:58.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:58 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:36:58.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:58 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:36:58.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:58 smithi114 ceph-mon[28260]: Manager daemon smithi114.rvhgji is now available 2024-09-20T19:36:58.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:58 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:36:58.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:58 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.rvhgji/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:36:58.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:36:58 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.rvhgji/trash_purge_schedule"}]: dispatch 2024-09-20T19:36:59.942 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-20T19:36:59.942 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-20T19:36:59.942 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-20T19:36:59.942 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-20T19:36:59.942 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 9 is available 2024-09-20T19:36:59.942 INFO:teuthology.orchestra.run.smithi114.stdout:Generating a dashboard self-signed certificate... 2024-09-20T19:37:00.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:00 smithi114 ceph-mon[28260]: [20/Sep/2024:19:36:58] ENGINE Bus STARTING 2024-09-20T19:37:00.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:00 smithi114 ceph-mon[28260]: [20/Sep/2024:19:36:59] ENGINE Serving on http://172.21.15.114:8765 2024-09-20T19:37:00.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:00 smithi114 ceph-mon[28260]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-20T19:37:00.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:00 smithi114 ceph-mon[28260]: mgrmap e11: smithi114.rvhgji(active, since 1.00522s) 2024-09-20T19:37:00.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:00 smithi114 ceph-mon[28260]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-20T19:37:00.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:00 smithi114 ceph-mon[28260]: [20/Sep/2024:19:36:59] ENGINE Serving on https://172.21.15.114:7150 2024-09-20T19:37:00.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:00 smithi114 ceph-mon[28260]: [20/Sep/2024:19:36:59] ENGINE Bus STARTED 2024-09-20T19:37:00.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:00 smithi114 ceph-mon[28260]: [20/Sep/2024:19:36:59] ENGINE Error in HTTPServer.serve 2024-09-20T19:37:00.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Traceback (most recent call last): 2024-09-20T19:37:00.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:37:00.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:37:00.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:37:00.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: self._run(expiration_interval) 2024-09-20T19:37:00.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:37:00.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:37:00.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:37:00.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:37:00.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:37:00.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: s = self.context.wrap_socket( 2024-09-20T19:37:00.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:37:00.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: return self.sslsocket_class._create( 2024-09-20T19:37:00.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:37:00.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: self.do_handshake() 2024-09-20T19:37:00.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:37:00.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: self._sslobj.do_handshake() 2024-09-20T19:37:00.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:37:01.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-20T19:37:01.156 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial admin user... 2024-09-20T19:37:01.377 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:01 smithi114 ceph-mon[28260]: mgrmap e12: smithi114.rvhgji(active, since 2s) 2024-09-20T19:37:01.377 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:01 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:01.377 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:01 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:01.377 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:01 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:02.368 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:02 smithi114 ceph-mon[28260]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:37:02.369 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:02 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:02.369 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:02 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:02.654 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Hgo1B.vwc4QUchY9wWDIculvj5IqFNx0Tc4nwbJcdiaZ0gfJH8O4W", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726861022, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-20T19:37:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:Fetching dashboard port number... 2024-09-20T19:37:03.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:03 smithi114 ceph-mon[28260]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:37:03.924 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 8443 2024-09-20T19:37:03.932 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T19:37:03.932 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-20T19:37:03.939 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T19:37:03.939 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-20T19:37:03.940 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-20T19:37:03.940 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-20T19:37:03.941 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph Dashboard is now available at: 2024-09-20T19:37:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:37:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: URL: https://smithi114.front.sepia.ceph.com:8443/ 2024-09-20T19:37:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: User: admin 2024-09-20T19:37:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: Password: lxgaw1sapn 2024-09-20T19:37:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:37:03.941 INFO:teuthology.orchestra.run.smithi114.stdout:Saving cluster configuration to /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config directory 2024-09-20T19:37:04.170 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:04 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1369910457' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-20T19:37:05.294 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:05 smithi114 ceph-mon[28260]: mgrmap e13: smithi114.rvhgji(active, since 6s) 2024-09-20T19:37:05.294 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:05 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/27285894' entity='client.admin' 2024-09-20T19:37:05.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-20T19:37:05.297 INFO:teuthology.orchestra.run.smithi114.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-20T19:37:05.297 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:37:05.298 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-20T19:37:05.298 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:37:05.298 INFO:teuthology.orchestra.run.smithi114.stdout:Or, if you are only running a single cluster on this host: 2024-09-20T19:37:05.298 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:37:05.298 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-20T19:37:05.298 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:37:05.298 INFO:teuthology.orchestra.run.smithi114.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-20T19:37:05.298 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:37:05.298 INFO:teuthology.orchestra.run.smithi114.stdout: ceph telemetry on 2024-09-20T19:37:05.298 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:37:05.298 INFO:teuthology.orchestra.run.smithi114.stdout:For more information see: 2024-09-20T19:37:05.298 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:37:05.298 INFO:teuthology.orchestra.run.smithi114.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-20T19:37:05.298 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:37:05.299 INFO:teuthology.orchestra.run.smithi114.stdout:Bootstrap complete. 2024-09-20T19:37:05.340 INFO:tasks.cephadm:Fetching config... 2024-09-20T19:37:05.340 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:37:05.340 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-20T19:37:05.356 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-20T19:37:05.357 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:37:05.357 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-20T19:37:05.412 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-20T19:37:05.412 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:37:05.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/keyring of=/dev/stdout 2024-09-20T19:37:05.476 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-20T19:37:05.476 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:37:05.477 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-20T19:37:05.532 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-20T19:37:05.533 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC83OTk+RSChAQKaDsLmkRr7EmZt7IHbiPRbgHW8gJWdT+W7ZmRhXQwws2B0M9tAbX5wxq48W/o1O9BhxbTUZG1Wgav4pbi4qPNdIfhcabDP496RUUzXkvTkBEEQeLcOcYAIfevCwqiIndeNXHxdRM3uQNPHd3P4CHG8IxCb9VE6edFMKZQ1UGATBEl5y5rryuRNjWESw1U2Cqi64VaKkjBAsFFgxLT8m0+uvY2SPpEW31lPYDmp3zEuPBW/f7Wf6MLYojNER43kLvpM8oNG6IVGL4rl867634B9lc5KrQ9cOp5sVfW5TCl6pGFUZ1awo8dMfSzIOFImaKBtiDupfe08xy5av/GSp+sWTUE/vJBlftgobdut5RfAd5Ab1LpAGikYNInfBNgosvHy1itAfRoCgo1boHc5FQfavkbwJpeG1xpR1k98i7/euhgX4LX947KZMYmWXJn11NiOC/TKgrsxBBFCh0T/I63dxtqmXqBPRAtssoFOZIv/2Wzvum1xfs= ceph-6a49c7bc-7787-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T19:37:05.606 INFO:teuthology.orchestra.run.smithi114.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC83OTk+RSChAQKaDsLmkRr7EmZt7IHbiPRbgHW8gJWdT+W7ZmRhXQwws2B0M9tAbX5wxq48W/o1O9BhxbTUZG1Wgav4pbi4qPNdIfhcabDP496RUUzXkvTkBEEQeLcOcYAIfevCwqiIndeNXHxdRM3uQNPHd3P4CHG8IxCb9VE6edFMKZQ1UGATBEl5y5rryuRNjWESw1U2Cqi64VaKkjBAsFFgxLT8m0+uvY2SPpEW31lPYDmp3zEuPBW/f7Wf6MLYojNER43kLvpM8oNG6IVGL4rl867634B9lc5KrQ9cOp5sVfW5TCl6pGFUZ1awo8dMfSzIOFImaKBtiDupfe08xy5av/GSp+sWTUE/vJBlftgobdut5RfAd5Ab1LpAGikYNInfBNgosvHy1itAfRoCgo1boHc5FQfavkbwJpeG1xpR1k98i7/euhgX4LX947KZMYmWXJn11NiOC/TKgrsxBBFCh0T/I63dxtqmXqBPRAtssoFOZIv/2Wzvum1xfs= ceph-6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:37:05.618 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC83OTk+RSChAQKaDsLmkRr7EmZt7IHbiPRbgHW8gJWdT+W7ZmRhXQwws2B0M9tAbX5wxq48W/o1O9BhxbTUZG1Wgav4pbi4qPNdIfhcabDP496RUUzXkvTkBEEQeLcOcYAIfevCwqiIndeNXHxdRM3uQNPHd3P4CHG8IxCb9VE6edFMKZQ1UGATBEl5y5rryuRNjWESw1U2Cqi64VaKkjBAsFFgxLT8m0+uvY2SPpEW31lPYDmp3zEuPBW/f7Wf6MLYojNER43kLvpM8oNG6IVGL4rl867634B9lc5KrQ9cOp5sVfW5TCl6pGFUZ1awo8dMfSzIOFImaKBtiDupfe08xy5av/GSp+sWTUE/vJBlftgobdut5RfAd5Ab1LpAGikYNInfBNgosvHy1itAfRoCgo1boHc5FQfavkbwJpeG1xpR1k98i7/euhgX4LX947KZMYmWXJn11NiOC/TKgrsxBBFCh0T/I63dxtqmXqBPRAtssoFOZIv/2Wzvum1xfs= ceph-6a49c7bc-7787-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T19:37:05.740 INFO:teuthology.orchestra.run.smithi160.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC83OTk+RSChAQKaDsLmkRr7EmZt7IHbiPRbgHW8gJWdT+W7ZmRhXQwws2B0M9tAbX5wxq48W/o1O9BhxbTUZG1Wgav4pbi4qPNdIfhcabDP496RUUzXkvTkBEEQeLcOcYAIfevCwqiIndeNXHxdRM3uQNPHd3P4CHG8IxCb9VE6edFMKZQ1UGATBEl5y5rryuRNjWESw1U2Cqi64VaKkjBAsFFgxLT8m0+uvY2SPpEW31lPYDmp3zEuPBW/f7Wf6MLYojNER43kLvpM8oNG6IVGL4rl867634B9lc5KrQ9cOp5sVfW5TCl6pGFUZ1awo8dMfSzIOFImaKBtiDupfe08xy5av/GSp+sWTUE/vJBlftgobdut5RfAd5Ab1LpAGikYNInfBNgosvHy1itAfRoCgo1boHc5FQfavkbwJpeG1xpR1k98i7/euhgX4LX947KZMYmWXJn11NiOC/TKgrsxBBFCh0T/I63dxtqmXqBPRAtssoFOZIv/2Wzvum1xfs= ceph-6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:37:05.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-20T19:37:06.157 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:37:07.757 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-20T19:37:07.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-20T19:37:07.955 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:07 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1755397463' entity='client.admin' 2024-09-20T19:37:08.180 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:37:09.932 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi160 2024-09-20T19:37:09.932 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-20T19:37:09.932 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.conf 2024-09-20T19:37:09.950 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-20T19:37:09.950 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:37:10.009 INFO:tasks.cephadm:Adding host smithi160 to orchestrator... 2024-09-20T19:37:10.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph orch host add smithi160 2024-09-20T19:37:10.214 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:09 smithi114 ceph-mon[28260]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:37:10.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:09 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:10.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:09 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:10.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:09 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:10.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:09 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:37:10.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:09 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:37:10.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:09 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:37:10.261 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:37:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:11 smithi114 ceph-mon[28260]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-20T19:37:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:11 smithi114 ceph-mon[28260]: Updating smithi114:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:37:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:11 smithi114 ceph-mon[28260]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:37:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:11 smithi114 ceph-mon[28260]: Updating smithi114:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:37:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:11 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:11 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:11 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:11 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi114", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-20T19:37:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:11 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi114", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-09-20T19:37:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:11 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:37:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:11 smithi114 ceph-mon[28260]: Deploying daemon ceph-exporter.smithi114 on smithi114 2024-09-20T19:37:12.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:12 smithi114 ceph-mon[28260]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:37:13.714 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:13 smithi114 ceph-mon[28260]: Deploying cephadm binary to smithi160 2024-09-20T19:37:14.162 INFO:teuthology.orchestra.run.smithi114.stdout:Added host 'smithi160' with addr '172.21.15.160' 2024-09-20T19:37:14.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:14 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:14.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:14 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:14.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:14 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:14.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:14 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:14.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:14 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi114", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T19:37:14.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:14 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi114", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-09-20T19:37:14.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:14 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:37:14.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:14 smithi114 ceph-mon[28260]: Deploying daemon crash.smithi114 on smithi114 2024-09-20T19:37:14.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:14 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:14.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:14 smithi114 ceph-mon[28260]: Added host smithi160 2024-09-20T19:37:14.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-20T19:37:15.353 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:37:15.965 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:37:15.965 INFO:teuthology.orchestra.run.smithi114.stdout:[{"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}, {"addr": "172.21.15.160", "hostname": "smithi160", "labels": [], "status": ""}] 2024-09-20T19:37:16.702 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:16 smithi114 ceph-mon[28260]: from='client.14187 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:37:16.734 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-20T19:37:16.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd crush tunables default 2024-09-20T19:37:17.021 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:37:17.995 INFO:teuthology.orchestra.run.smithi114.stderr:adjusted tunables profile to default 2024-09-20T19:37:18.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:17 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:18.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:17 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:18.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:17 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:18.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:17 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:18.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:17 smithi114 ceph-mon[28260]: Deploying daemon node-exporter.smithi114 on smithi114 2024-09-20T19:37:18.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:17 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1731597838' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-20T19:37:18.632 INFO:tasks.cephadm:Adding mon.smithi114 on smithi114 2024-09-20T19:37:18.633 INFO:tasks.cephadm:Adding mon.smithi160 on smithi160 2024-09-20T19:37:18.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph orch apply mon '2;smithi114:172.21.15.114=smithi114;smithi160:172.21.15.160=smithi160' 2024-09-20T19:37:18.941 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:18.985 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:19.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:18 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1731597838' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T19:37:19.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:18 smithi114 ceph-mon[28260]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:37:19.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:18 smithi114 ceph-mon[28260]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:19.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:18 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:19.570 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled mon update... 2024-09-20T19:37:20.152 DEBUG:teuthology.orchestra.run.smithi160:mon.smithi160> sudo journalctl -f -n 0 -u ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@mon.smithi160.service 2024-09-20T19:37:20.155 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:20.155 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:20.426 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:20.470 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:20.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:20 smithi114 ceph-mon[28260]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi114:172.21.15.114=smithi114;smithi160:172.21.15.160=smithi160", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:37:20.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:20 smithi114 ceph-mon[28260]: Saving service mon spec with placement smithi114:172.21.15.114=smithi114;smithi160:172.21.15.160=smithi160;count:2 2024-09-20T19:37:20.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:20 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:20.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:20 smithi114 ceph-mon[28260]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:21.123 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:37:21.123 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:37:21.123 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:37:21.714 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:21 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/935441595' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:37:21.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:21 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:21.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:21 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:21.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:21 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:21.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:21 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:22.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:22.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:22.953 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:22.964 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:22 smithi114 ceph-mon[28260]: Deploying daemon alertmanager.smithi114 on smithi114 2024-09-20T19:37:22.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:22 smithi114 ceph-mon[28260]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:22.997 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:23.801 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:37:23.801 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:37:23.801 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:37:24.464 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:24 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:24.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:24 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/1474367556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:37:25.464 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:25 smithi114 ceph-mon[28260]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:25.575 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:25.576 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:25.821 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:25.864 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:26.488 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:37:26.488 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:37:26.488 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:37:27.448 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:27 smithi114 ceph-mon[28260]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:27.448 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:27 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/3757862911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:37:28.053 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:28.053 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:28.300 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:28.343 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:28.967 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:37:28.968 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:37:28.968 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:37:29.258 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:29 smithi114 ceph-mon[28260]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:29.258 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:29 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/454536901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:37:30.566 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:30.566 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:30.809 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:30.853 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:30.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:30 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:30.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:30 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:30.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:30 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:30.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:30 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:30.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:30 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:30.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:30 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:30.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:30 smithi114 ceph-mon[28260]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T19:37:30.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:30 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:30.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:30 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:30.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:30 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:37:30.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:30 smithi114 ceph-mon[28260]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:37:30.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:30 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:30.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:30 smithi114 ceph-mon[28260]: Deploying daemon grafana.smithi114 on smithi114 2024-09-20T19:37:30.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:30 smithi114 ceph-mon[28260]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:31.491 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:37:31.491 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:37:31.491 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:37:31.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:31 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/273866719' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:37:32.964 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:32 smithi114 ceph-mon[28260]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:33.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:33.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:33.270 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:33.315 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:33.929 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:37:33.930 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:37:33.930 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:37:34.464 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:34 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:34.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:34 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/2337789810' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:37:35.464 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:35 smithi114 ceph-mon[28260]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:35.486 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:35.486 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:35.729 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:35.773 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:36.411 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:37:36.411 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:37:36.411 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:37:37.434 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:37 smithi114 ceph-mon[28260]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:37.434 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:37 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/2744503246' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:37:37.998 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:37.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:38.235 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:38.280 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:38.927 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:37:38.927 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:37:38.927 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:37:39.364 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:39 smithi114 ceph-mon[28260]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:39.364 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:39 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/2634639147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:37:40.507 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:40.507 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:40.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:40 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:40.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:40 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:40.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:40 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:40.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:40 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:40.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:40 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:40.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:40 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:40.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:40 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:40.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:40 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:40.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:40 smithi114 ceph-mon[28260]: Deploying daemon prometheus.smithi114 on smithi114 2024-09-20T19:37:40.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:40 smithi114 ceph-mon[28260]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:40.762 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:40.806 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:41.462 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:37:41.462 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:37:41.463 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:37:41.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:41 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/2261191567' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:37:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:42 smithi114 ceph-mon[28260]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:43.120 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:43.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:43.361 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:43.406 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:44.034 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:37:44.034 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:37:44.034 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:37:44.464 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:44 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:44.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:44 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/2480807244' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:37:45.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:45 smithi114 ceph-mon[28260]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:45.632 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:45.632 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:45.875 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:45.918 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:46.522 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:37:46.522 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:37:46.522 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:37:47.464 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:47 smithi114 ceph-mon[28260]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:47.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:47 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/2972851997' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:37:48.122 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:48.125 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:48.382 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:48.427 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:49.044 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:37:49.044 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:37:49.045 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:37:49.317 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:49 smithi114 ceph-mon[28260]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:49.317 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:49 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/3854114161' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:37:50.625 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:50.625 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:50.869 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:50.912 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:51.452 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:51 smithi114 ceph-mon[28260]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:37:51.452 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:51 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:51.452 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:51 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:51.452 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:51 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:51.453 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:51 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T19:37:51.585 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:37:51.585 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:37:51.585 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:37:52.692 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:52 smithi114 ceph-mon[28260]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T19:37:52.693 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:52 smithi114 ceph-mon[28260]: mgrmap e14: smithi114.rvhgji(active, since 53s) 2024-09-20T19:37:52.693 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:52 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/1676733779' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:37:53.175 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:53.175 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:53.418 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:53.472 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:54.064 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:37:54.064 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:37:54.064 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:37:54.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:54 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/1428509530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:37:55.727 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:55.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:55.970 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:56.023 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:56.503 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:56 smithi114 ceph-mon[28260]: Active manager daemon smithi114.rvhgji restarted 2024-09-20T19:37:56.503 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:56 smithi114 ceph-mon[28260]: Activating manager daemon smithi114.rvhgji 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:56 smithi114 ceph-mon[28260]: osdmap e5: 0 total, 0 up, 0 in 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:56 smithi114 ceph-mon[28260]: mgrmap e15: smithi114.rvhgji(active, starting, since 0.00552936s) 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:56 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:56 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mgr metadata", "who": "smithi114.rvhgji", "id": "smithi114.rvhgji"}]: dispatch 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:56 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:56 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:56 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:56 smithi114 ceph-mon[28260]: Manager daemon smithi114.rvhgji is now available 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:56 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:56 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:56 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:37:56.504 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:56 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.rvhgji/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:37:56.711 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:37:56.711 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:37:56.711 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:37:57.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:57 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.rvhgji/trash_purge_schedule"}]: dispatch 2024-09-20T19:37:57.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:57 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/1430781939' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:37:57.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:57 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:57.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:57 smithi114 ceph-mon[28260]: mgrmap e16: smithi114.rvhgji(active, since 1.00811s) 2024-09-20T19:37:58.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:37:58.366 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:37:58.612 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:58.692 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:58 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:58.692 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:58 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:37:59.171 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T19:37:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:59 smithi114 ceph-mon[28260]: [20/Sep/2024:19:37:57] ENGINE Bus STARTING 2024-09-20T19:37:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:59 smithi114 ceph-mon[28260]: [20/Sep/2024:19:37:57] ENGINE Serving on http://172.21.15.114:8765 2024-09-20T19:37:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:59 smithi114 ceph-mon[28260]: [20/Sep/2024:19:37:57] ENGINE Serving on https://172.21.15.114:7150 2024-09-20T19:37:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:59 smithi114 ceph-mon[28260]: [20/Sep/2024:19:37:57] ENGINE Bus STARTED 2024-09-20T19:37:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:59 smithi114 ceph-mon[28260]: [20/Sep/2024:19:37:57] ENGINE Error in HTTPServer.serve 2024-09-20T19:37:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Traceback (most recent call last): 2024-09-20T19:37:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:37:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:37:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:37:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: self._run(expiration_interval) 2024-09-20T19:37:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:37:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:37:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:37:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:37:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:37:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: s = self.context.wrap_socket( 2024-09-20T19:37:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:37:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: return self.sslsocket_class._create( 2024-09-20T19:37:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:37:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: self.do_handshake() 2024-09-20T19:37:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:37:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: self._sslobj.do_handshake() 2024-09-20T19:37:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:37:59.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:37:59 smithi114 ceph-mon[28260]: mgrmap e17: smithi114.rvhgji(active, since 2s) 2024-09-20T19:38:00.165 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:38:00.166 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:38:00.166 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:38:00.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:00 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/2531550656' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:00.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:00 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:00.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:00 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:02.060 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:38:02.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:38:02.304 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:38:02.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:01 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:02.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:01 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:02.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:01 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:38:02.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:01 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:02.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:01 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:02.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:01 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:02.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:01 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:02.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:01 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:38:02.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:01 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:02.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:01 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:38:03.073 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:38:03.074 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:38:03.074 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:38:03.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-20T19:38:03.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-20T19:38:03.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: Updating smithi160:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:38:03.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: Updating smithi114:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:38:03.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:38:03.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:38:03.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: Updating smithi160:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:38:03.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: Updating smithi114:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:38:03.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:03.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:03.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:03.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:03.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:03.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi160", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-20T19:38:03.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi160", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-09-20T19:38:03.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:03.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:03 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/3652151039' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:04.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:04 smithi114 ceph-mon[28260]: Deploying daemon ceph-exporter.smithi160 on smithi160 2024-09-20T19:38:05.383 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:38:05.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:38:05.642 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:38:06.325 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:38:06.325 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:38:06.325 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:38:06.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:06 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:06.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:06 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:06.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:06 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:06.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:06 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:06.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:06 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi160", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T19:38:06.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:06 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi160", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-09-20T19:38:06.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:06 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:06.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:06 smithi114 ceph-mon[28260]: Deploying daemon crash.smithi160 on smithi160 2024-09-20T19:38:06.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:06 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:06.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:06 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/1369108929' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:08.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:38:08.269 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:38:08.701 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:38:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:38:09.438 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:38:09.438 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:38:09.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:09 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:09.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:09 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:09.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:09 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:09.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:09 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:09.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:09 smithi114 ceph-mon[28260]: Deploying daemon node-exporter.smithi160 on smithi160 2024-09-20T19:38:09.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:09 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/3595498580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:11.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:38:11.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:38:11.865 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:38:12.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:12 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:12.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:12 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:38:12.740 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:38:12.741 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:38:12.741 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:38:13.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:13 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/3063571765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:14.348 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:38:14.348 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:38:14.598 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:38:14.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:14 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:14.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:14 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:14.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:14 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:14.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:14 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:14.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:14 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi160.bspiak", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:38:14.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:14 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi160.bspiak", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T19:38:14.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:14 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:38:14.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:14 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:14.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:14 smithi114 ceph-mon[28260]: Deploying daemon mgr.smithi160.bspiak on smithi160 2024-09-20T19:38:15.863 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:38:15.863 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:38:15.863 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:38:16.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:15 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/811366821' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:17.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:17 smithi114 ceph-mon[28260]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:17.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:17 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:17.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:17 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:17.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:17 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:17.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:17 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:17.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:17 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:17.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:17 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:38:17.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:17 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:17.929 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:38:17.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:38:18.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:18 smithi114 ceph-mon[28260]: Deploying daemon mon.smithi160 on smithi160 2024-09-20T19:38:19.147 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi160/config 2024-09-20T19:38:19.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:19 smithi114 ceph-mon[28260]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:19.991 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:38:19.991 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:35:54.449562Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T19:38:19.991 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-20T19:38:20.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:20 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/44057859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.762 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.14187 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:38:20.762 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.762 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.762 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.763 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.763 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Deploying daemon node-exporter.smithi114 on smithi114 2024-09-20T19:38:20.763 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1731597838' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-20T19:38:20.763 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1731597838' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T19:38:20.763 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T19:38:20.763 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.763 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.763 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi114:172.21.15.114=smithi114;smithi160:172.21.15.160=smithi160", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:38:20.763 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Saving service mon spec with placement smithi114:172.21.15.114=smithi114;smithi160:172.21.15.160=smithi160;count:2 2024-09-20T19:38:20.763 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.763 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.763 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/935441595' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.763 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.764 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.764 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.764 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.764 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Deploying daemon alertmanager.smithi114 on smithi114 2024-09-20T19:38:20.764 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.764 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.764 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/1474367556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.764 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.764 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.764 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/3757862911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.764 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.764 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/454536901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.764 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.765 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.765 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.766 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.766 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.766 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.766 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T19:38:20.766 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.766 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.766 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:38:20.766 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T19:38:20.766 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.766 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Deploying daemon grafana.smithi114 on smithi114 2024-09-20T19:38:20.766 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.767 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/273866719' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.767 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.767 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.767 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/2337789810' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.767 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.767 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.767 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/2744503246' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.767 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.767 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/2634639147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.767 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.767 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.767 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.768 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.768 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.768 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.768 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.768 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.768 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Deploying daemon prometheus.smithi114 on smithi114 2024-09-20T19:38:20.768 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.768 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/2261191567' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.768 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.768 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.768 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/2480807244' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.768 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.768 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.768 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/2972851997' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.769 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.769 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/3854114161' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.769 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.769 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.769 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.769 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.769 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T19:38:20.769 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14160 172.21.15.114:0/4064540052' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T19:38:20.769 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: mgrmap e14: smithi114.rvhgji(active, since 53s) 2024-09-20T19:38:20.769 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/1676733779' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.770 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/1428509530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.770 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Active manager daemon smithi114.rvhgji restarted 2024-09-20T19:38:20.770 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Activating manager daemon smithi114.rvhgji 2024-09-20T19:38:20.770 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: osdmap e5: 0 total, 0 up, 0 in 2024-09-20T19:38:20.770 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: mgrmap e15: smithi114.rvhgji(active, starting, since 0.00552936s) 2024-09-20T19:38:20.770 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-09-20T19:38:20.770 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mgr metadata", "who": "smithi114.rvhgji", "id": "smithi114.rvhgji"}]: dispatch 2024-09-20T19:38:20.770 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T19:38:20.770 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T19:38:20.770 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T19:38:20.770 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Manager daemon smithi114.rvhgji is now available 2024-09-20T19:38:20.770 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.770 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.771 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:38:20.771 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.rvhgji/mirror_snapshot_schedule"}]: dispatch 2024-09-20T19:38:20.771 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.rvhgji/trash_purge_schedule"}]: dispatch 2024-09-20T19:38:20.771 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/1430781939' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.771 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.771 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: mgrmap e16: smithi114.rvhgji(active, since 1.00811s) 2024-09-20T19:38:20.771 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.771 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.771 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: [20/Sep/2024:19:37:57] ENGINE Bus STARTING 2024-09-20T19:38:20.771 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: [20/Sep/2024:19:37:57] ENGINE Serving on http://172.21.15.114:8765 2024-09-20T19:38:20.771 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: [20/Sep/2024:19:37:57] ENGINE Serving on https://172.21.15.114:7150 2024-09-20T19:38:20.771 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: [20/Sep/2024:19:37:57] ENGINE Bus STARTED 2024-09-20T19:38:20.771 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: [20/Sep/2024:19:37:57] ENGINE Error in HTTPServer.serve 2024-09-20T19:38:20.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Traceback (most recent call last): 2024-09-20T19:38:20.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T19:38:20.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: self._connections.run(self.expiration_interval) 2024-09-20T19:38:20.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T19:38:20.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: self._run(expiration_interval) 2024-09-20T19:38:20.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T19:38:20.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T19:38:20.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T19:38:20.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T19:38:20.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T19:38:20.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: s = self.context.wrap_socket( 2024-09-20T19:38:20.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T19:38:20.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: return self.sslsocket_class._create( 2024-09-20T19:38:20.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T19:38:20.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: self.do_handshake() 2024-09-20T19:38:20.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T19:38:20.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: self._sslobj.do_handshake() 2024-09-20T19:38:20.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T19:38:20.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: mgrmap e17: smithi114.rvhgji(active, since 2s) 2024-09-20T19:38:20.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/2531550656' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:38:20.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:38:20.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:20.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:38:20.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-20T19:38:20.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-20T19:38:20.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Updating smithi160:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:38:20.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Updating smithi114:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:38:20.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:38:20.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:38:20.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Updating smithi160:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:38:20.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Updating smithi114:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T19:38:20.775 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.775 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.775 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.775 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.775 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.775 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi160", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-20T19:38:20.775 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi160", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-09-20T19:38:20.775 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:20.775 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/3652151039' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.775 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Deploying daemon ceph-exporter.smithi160 on smithi160 2024-09-20T19:38:20.775 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.775 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.775 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.775 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.775 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi160", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T19:38:20.776 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi160", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-09-20T19:38:20.776 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:20.776 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Deploying daemon crash.smithi160 on smithi160 2024-09-20T19:38:20.776 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.776 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/1369108929' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.776 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.776 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.776 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.776 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.776 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Deploying daemon node-exporter.smithi160 on smithi160 2024-09-20T19:38:20.776 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/3595498580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.776 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.777 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:38:20.777 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/3063571765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.777 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.777 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.777 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.777 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.777 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi160.bspiak", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:38:20.777 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi160.bspiak", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T19:38:20.777 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:38:20.777 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:20.777 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Deploying daemon mgr.smithi160.bspiak on smithi160 2024-09-20T19:38:20.777 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/811366821' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.777 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.777 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.778 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.778 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.778 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.778 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:20.778 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:38:20.778 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:20.778 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: Deploying daemon mon.smithi160 on smithi160 2024-09-20T19:38:20.778 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:20.778 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/44057859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:20.778 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 ceph-mon[35859]: mon.smithi160@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-09-20T19:38:21.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 bash[35824]: 961cc10b05ccb90feea6c935996f5cbd1b395e7e82062b23df47686b3ddc5227 2024-09-20T19:38:21.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:20 smithi160 systemd[1]: Started Ceph mon.smithi160 for 6a49c7bc-7787-11ef-baf5-efdc52797490. 2024-09-20T19:38:21.575 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T19:38:21.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T19:38:21.819 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi160/config 2024-09-20T19:38:26.067 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:38:26.067 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":2,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","modified":"2024-09-20T19:38:20.766583Z","created":"2024-09-20T19:35:54.449562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi160","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:3300","nonce":0},{"type":"v1","addr":"172.21.15.160:6789","nonce":0}]},"addr":"172.21.15.160:6789/0","public_addr":"172.21.15.160:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-20T19:38:26.067 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 2 2024-09-20T19:38:26.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-09-20T19:38:26.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: mon.smithi114 calling monitor election 2024-09-20T19:38:26.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:38:26.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.? 172.21.15.160:0/726735000' entity='mgr.smithi160.bspiak' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi160.bspiak/crt"}]: dispatch 2024-09-20T19:38:26.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:38:26.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:26.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:38:26.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: mon.smithi160 calling monitor election 2024-09-20T19:38:26.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:38:26.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:26.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:38:26.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:38:26.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: mon.smithi114 is new leader, mons smithi114,smithi160 in quorum (ranks 0,1) 2024-09-20T19:38:26.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: monmap epoch 2 2024-09-20T19:38:26.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:38:26.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: last_changed 2024-09-20T19:38:20.766583+0000 2024-09-20T19:38:26.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: created 2024-09-20T19:35:54.449562+0000 2024-09-20T19:38:26.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: min_mon_release 19 (squid) 2024-09-20T19:38:26.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: election_strategy: 1 2024-09-20T19:38:26.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.smithi114 2024-09-20T19:38:26.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.smithi160 2024-09-20T19:38:26.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: fsmap 2024-09-20T19:38:26.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: osdmap e5: 0 total, 0 up, 0 in 2024-09-20T19:38:26.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: mgrmap e17: smithi114.rvhgji(active, since 29s) 2024-09-20T19:38:26.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: overall HEALTH_OK 2024-09-20T19:38:26.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: Standby manager daemon smithi160.bspiak started 2024-09-20T19:38:26.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:26.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.? 172.21.15.160:0/726735000' entity='mgr.smithi160.bspiak' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T19:38:26.218 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:26.218 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.? 172.21.15.160:0/726735000' entity='mgr.smithi160.bspiak' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi160.bspiak/key"}]: dispatch 2024-09-20T19:38:26.218 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.? 172.21.15.160:0/726735000' entity='mgr.smithi160.bspiak' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T19:38:26.218 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:26.218 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:26.218 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:26.218 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:26.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-09-20T19:38:26.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: mon.smithi114 calling monitor election 2024-09-20T19:38:26.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:38:26.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.? 172.21.15.160:0/726735000' entity='mgr.smithi160.bspiak' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi160.bspiak/crt"}]: dispatch 2024-09-20T19:38:26.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:38:26.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:26.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:38:26.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: mon.smithi160 calling monitor election 2024-09-20T19:38:26.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:38:26.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:26.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:38:26.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:38:26.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: mon.smithi114 is new leader, mons smithi114,smithi160 in quorum (ranks 0,1) 2024-09-20T19:38:26.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: monmap epoch 2 2024-09-20T19:38:26.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:38:26.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: last_changed 2024-09-20T19:38:20.766583+0000 2024-09-20T19:38:26.281 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: created 2024-09-20T19:35:54.449562+0000 2024-09-20T19:38:26.281 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: min_mon_release 19 (squid) 2024-09-20T19:38:26.281 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: election_strategy: 1 2024-09-20T19:38:26.281 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.smithi114 2024-09-20T19:38:26.281 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.smithi160 2024-09-20T19:38:26.281 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: fsmap 2024-09-20T19:38:26.281 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: osdmap e5: 0 total, 0 up, 0 in 2024-09-20T19:38:26.281 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: mgrmap e17: smithi114.rvhgji(active, since 29s) 2024-09-20T19:38:26.281 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: overall HEALTH_OK 2024-09-20T19:38:26.281 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: Standby manager daemon smithi160.bspiak started 2024-09-20T19:38:26.281 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:26.281 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.? 172.21.15.160:0/726735000' entity='mgr.smithi160.bspiak' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T19:38:26.281 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:26.281 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.? 172.21.15.160:0/726735000' entity='mgr.smithi160.bspiak' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi160.bspiak/key"}]: dispatch 2024-09-20T19:38:26.281 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.? 172.21.15.160:0/726735000' entity='mgr.smithi160.bspiak' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T19:38:26.282 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:26.282 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:26.282 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:26.282 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:26.711 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-20T19:38:26.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config generate-minimal-conf 2024-09-20T19:38:26.972 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:27.022 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:26 smithi160 ceph-mon[35859]: mgrmap e18: smithi114.rvhgji(active, since 29s), standbys: smithi160.bspiak 2024-09-20T19:38:27.022 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:26 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mgr metadata", "who": "smithi160.bspiak", "id": "smithi160.bspiak"}]: dispatch 2024-09-20T19:38:27.022 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:26 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:38:27.023 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:26 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/2288636872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:27.023 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:26 smithi160 ceph-mon[35859]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:27.023 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:26 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:38:27.023 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:26 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:38:27.093 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:26 smithi114 ceph-mon[28260]: mgrmap e18: smithi114.rvhgji(active, since 29s), standbys: smithi160.bspiak 2024-09-20T19:38:27.093 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:26 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mgr metadata", "who": "smithi160.bspiak", "id": "smithi160.bspiak"}]: dispatch 2024-09-20T19:38:27.093 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:26 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:38:27.094 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:26 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/2288636872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T19:38:27.094 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:26 smithi114 ceph-mon[28260]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:27.094 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:26 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:38:27.094 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:26 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:38:27.598 INFO:teuthology.orchestra.run.smithi114.stdout:# minimal ceph.conf for 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:38:27.598 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-09-20T19:38:27.598 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T19:38:27.598 INFO:teuthology.orchestra.run.smithi114.stdout: mon_host = [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2024-09-20T19:38:27.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:27 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3561347793' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:28.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:27 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/3561347793' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:28.191 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-20T19:38:28.192 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:38:28.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T19:38:28.218 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:38:28.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:38:28.284 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-20T19:38:28.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T19:38:28.310 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-20T19:38:28.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T19:38:28.374 INFO:tasks.cephadm:setting config [client] client_mount_timeout = 600 2024-09-20T19:38:28.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set client client_mount_timeout 600 2024-09-20T19:38:28.377 INFO:tasks.cephadm:setting config [client] debug_client = 20 2024-09-20T19:38:28.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set client debug_client 20 2024-09-20T19:38:28.420 INFO:tasks.cephadm:setting config [client] debug_ms = 1 2024-09-20T19:38:28.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set client debug_ms 1 2024-09-20T19:38:28.463 INFO:tasks.cephadm:setting config [client] rados_mon_op_timeout = 900 2024-09-20T19:38:28.463 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set client rados_mon_op_timeout 900 2024-09-20T19:38:28.506 INFO:tasks.cephadm:setting config [client] rados_osd_op_timeout = 900 2024-09-20T19:38:28.506 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set client rados_osd_op_timeout 900 2024-09-20T19:38:28.549 INFO:tasks.cephadm:setting config [mds] debug_mds = 20 2024-09-20T19:38:28.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set mds debug_mds 20 2024-09-20T19:38:28.592 INFO:tasks.cephadm:setting config [mds] debug_mds_balancer = 20 2024-09-20T19:38:28.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set mds debug_mds_balancer 20 2024-09-20T19:38:28.636 INFO:tasks.cephadm:setting config [mds] debug_ms = 2 2024-09-20T19:38:28.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set mds debug_ms 2 2024-09-20T19:38:28.689 INFO:tasks.cephadm:setting config [mds] mds_debug_frag = True 2024-09-20T19:38:28.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set mds mds_debug_frag True 2024-09-20T19:38:28.735 INFO:tasks.cephadm:setting config [mds] mds_debug_scatterstat = True 2024-09-20T19:38:28.735 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set mds mds_debug_scatterstat True 2024-09-20T19:38:28.737 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:28.739 INFO:tasks.cephadm:setting config [mds] mds_op_complaint_time = 180 2024-09-20T19:38:28.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set mds mds_op_complaint_time 180 2024-09-20T19:38:28.788 INFO:tasks.cephadm:setting config [mds] mds_verify_scatter = True 2024-09-20T19:38:28.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set mds mds_verify_scatter True 2024-09-20T19:38:28.832 INFO:tasks.cephadm:setting config [mds] osd_op_complaint_time = 180 2024-09-20T19:38:28.833 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set mds osd_op_complaint_time 180 2024-09-20T19:38:28.868 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:28.870 INFO:tasks.cephadm:setting config [mds] rados_mon_op_timeout = 900 2024-09-20T19:38:28.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set mds rados_mon_op_timeout 900 2024-09-20T19:38:28.919 INFO:tasks.cephadm:setting config [mds] rados_osd_op_timeout = 900 2024-09-20T19:38:28.920 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set mds rados_osd_op_timeout 900 2024-09-20T19:38:28.978 INFO:tasks.cephadm:setting config [mon] mon_op_complaint_time = 120 2024-09-20T19:38:28.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set mon mon_op_complaint_time 120 2024-09-20T19:38:29.024 INFO:tasks.cephadm:setting config [mon] mon_warn_on_pool_no_app = False 2024-09-20T19:38:29.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set mon mon_warn_on_pool_no_app False 2024-09-20T19:38:29.067 INFO:tasks.cephadm:setting config [osd] osd_op_complaint_time = 180 2024-09-20T19:38:29.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config set osd osd_op_complaint_time 180 2024-09-20T19:38:29.111 DEBUG:tasks.cephadm:set 18 configs 2024-09-20T19:38:29.111 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi114') 2024-09-20T19:38:29.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:29 smithi160 ceph-mon[35859]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:29.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:29 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:29.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:29 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:29.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:29 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:29.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:29 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:38:29.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:29 smithi160 ceph-mon[35859]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-20T19:38:29.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:29 smithi160 ceph-mon[35859]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-20T19:38:29.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:29 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:29.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:29 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:29.596 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:29 smithi114 ceph-mon[28260]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:29.596 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:29 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:29.596 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:29 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:29.596 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:29 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:29.596 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:29 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:38:29.597 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:29 smithi114 ceph-mon[28260]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-20T19:38:29.597 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:29 smithi114 ceph-mon[28260]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-20T19:38:29.597 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:29 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:29.597 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:29 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:30.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:30 smithi114 ceph-mon[28260]: Updating smithi160:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:38:30.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:30 smithi114 ceph-mon[28260]: Updating smithi114:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:38:30.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:30 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:30.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:30 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:30.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:30 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:30.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:30 smithi114 ceph-mon[28260]: Reconfiguring mon.smithi114 (unknown last config time)... 2024-09-20T19:38:30.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:30 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:38:30.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:30 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:38:30.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:30 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:30.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:30 smithi114 ceph-mon[28260]: Reconfiguring daemon mon.smithi114 on smithi114 2024-09-20T19:38:30.544 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:30 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/538015668' entity='client.admin' 2024-09-20T19:38:30.544 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:30 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3734371332' entity='client.admin' 2024-09-20T19:38:30.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:30 smithi160 ceph-mon[35859]: Updating smithi160:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:38:30.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:30 smithi160 ceph-mon[35859]: Updating smithi114:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T19:38:30.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:30 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:30.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:30 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:30.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:30 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:30.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:30 smithi160 ceph-mon[35859]: Reconfiguring mon.smithi114 (unknown last config time)... 2024-09-20T19:38:30.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:30 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:38:30.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:30 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:38:30.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:30 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:30.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:30 smithi160 ceph-mon[35859]: Reconfiguring daemon mon.smithi114 on smithi114 2024-09-20T19:38:30.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:30 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/538015668' entity='client.admin' 2024-09-20T19:38:30.781 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:30 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/3734371332' entity='client.admin' 2024-09-20T19:38:31.410 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:31 smithi114 ceph-mon[28260]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:31.410 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:31 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:31.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:31 smithi160 ceph-mon[35859]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:31.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:31 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:31.965 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:32.378 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi114') 2024-09-20T19:38:32.378 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi114') 2024-09-20T19:38:32.530 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:32.540 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:32.540 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:32.546 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:32.581 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:32.590 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:32.631 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:32.634 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:32.636 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:32.638 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:32.640 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:33.270 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:33.276 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:33.594 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:33 smithi114 ceph-mon[28260]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:33.594 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:33 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:33.594 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:33 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:33.594 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:33 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi114.rvhgji", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:38:33.594 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:33 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:38:33.594 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:33 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:33.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:33 smithi160 ceph-mon[35859]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:33.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:33 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:33.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:33 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:33.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:33 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi114.rvhgji", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:38:33.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:33 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:38:33.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:33 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:34.390 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:34 smithi114 ceph-mon[28260]: Reconfiguring mgr.smithi114.rvhgji (unknown last config time)... 2024-09-20T19:38:34.390 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:34 smithi114 ceph-mon[28260]: Reconfiguring daemon mgr.smithi114.rvhgji on smithi114 2024-09-20T19:38:34.390 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:34 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/4190419036' entity='client.admin' 2024-09-20T19:38:34.390 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:34 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/921610605' entity='client.admin' 2024-09-20T19:38:34.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:34 smithi160 ceph-mon[35859]: Reconfiguring mgr.smithi114.rvhgji (unknown last config time)... 2024-09-20T19:38:34.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:34 smithi160 ceph-mon[35859]: Reconfiguring daemon mgr.smithi114.rvhgji on smithi114 2024-09-20T19:38:34.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:34 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/4190419036' entity='client.admin' 2024-09-20T19:38:34.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:34 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/921610605' entity='client.admin' 2024-09-20T19:38:35.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:35 smithi114 ceph-mon[28260]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:35.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:35 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/266643831' entity='client.admin' 2024-09-20T19:38:35.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:35 smithi160 ceph-mon[35859]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:35.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:35 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/266643831' entity='client.admin' 2024-09-20T19:38:36.816 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:36 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3751157984' entity='client.admin' 2024-09-20T19:38:36.816 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:36 smithi114 ceph-mon[28260]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:36.816 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:36 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/538663038' entity='client.admin' 2024-09-20T19:38:36.948 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:36 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/3751157984' entity='client.admin' 2024-09-20T19:38:36.948 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:36 smithi160 ceph-mon[35859]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:36.948 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:36 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/538663038' entity='client.admin' 2024-09-20T19:38:39.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:39 smithi114 ceph-mon[28260]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:39.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:39 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/2058397997' entity='client.admin' 2024-09-20T19:38:39.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:39 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/754445192' entity='client.admin' 2024-09-20T19:38:39.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:39 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1792339823' entity='client.admin' 2024-09-20T19:38:39.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:39 smithi160 ceph-mon[35859]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:39.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:39 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/2058397997' entity='client.admin' 2024-09-20T19:38:39.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:39 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/754445192' entity='client.admin' 2024-09-20T19:38:39.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:39 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1792339823' entity='client.admin' 2024-09-20T19:38:40.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:40 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1299610677' entity='client.admin' 2024-09-20T19:38:40.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:40 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/2967630827' entity='client.admin' 2024-09-20T19:38:40.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:40 smithi160 ceph-mon[35859]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:40.819 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:40 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1299610677' entity='client.admin' 2024-09-20T19:38:40.819 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:40 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/2967630827' entity='client.admin' 2024-09-20T19:38:40.819 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:40 smithi114 ceph-mon[28260]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:42.150 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:41 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1099816704' entity='client.admin' 2024-09-20T19:38:42.150 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:41 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/2030933147' entity='client.admin' 2024-09-20T19:38:42.150 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:41 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:38:42.150 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:41 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/4203197906' entity='client.admin' 2024-09-20T19:38:42.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:41 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1099816704' entity='client.admin' 2024-09-20T19:38:42.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:41 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/2030933147' entity='client.admin' 2024-09-20T19:38:42.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:41 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:38:42.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:41 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/4203197906' entity='client.admin' 2024-09-20T19:38:43.149 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:42 smithi114 ceph-mon[28260]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:43.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:42 smithi160 ceph-mon[35859]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:44.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:44.545+0000 7fa37ad32640 1 -- 172.21.15.114:0/4020762463 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fa35c06e280 msgr2=0x7fa35c070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:38:44.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:44.545+0000 7fa37ad32640 1 --2- 172.21.15.114:0/4020762463 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fa35c06e280 0x7fa35c070740 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa36802f9b0 tx=0x7fa368009360 comp rx=0 tx=0).stop 2024-09-20T19:38:44.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:44.545+0000 7fa37ad32640 1 -- 172.21.15.114:0/4020762463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa374101960 msgr2=0x7fa3741973c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:38:44.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:44.545+0000 7fa37ad32640 1 --2- 172.21.15.114:0/4020762463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa374101960 0x7fa3741973c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa36400d8d0 tx=0x7fa36400dd90 comp rx=0 tx=0).stop 2024-09-20T19:38:44.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:44.545+0000 7fa37ad32640 1 -- 172.21.15.114:0/4020762463 shutdown_connections 2024-09-20T19:38:44.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:44.545+0000 7fa37ad32640 1 --2- 172.21.15.114:0/4020762463 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fa35c06e280 0x7fa35c070740 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:44.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:44.545+0000 7fa37ad32640 1 --2- 172.21.15.114:0/4020762463 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa374102b60 0x7fa374197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:44.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:44.545+0000 7fa37ad32640 1 --2- 172.21.15.114:0/4020762463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa374101960 0x7fa3741973c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:44.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:44.545+0000 7fa37ad32640 1 -- 172.21.15.114:0/4020762463 >> 172.21.15.114:0/4020762463 conn(0x7fa3740fd0f0 msgr2=0x7fa374106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:38:44.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:44.545+0000 7fa37ad32640 1 -- 172.21.15.114:0/4020762463 shutdown_connections 2024-09-20T19:38:44.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:44.545+0000 7fa37ad32640 1 -- 172.21.15.114:0/4020762463 wait complete. 2024-09-20T19:38:45.424 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:45.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:45 smithi114 ceph-mon[28260]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:45.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:45 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/4020762463' entity='client.admin' 2024-09-20T19:38:45.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:45 smithi160 ceph-mon[35859]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:45.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:45 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/4020762463' entity='client.admin' 2024-09-20T19:38:45.859 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi114') 2024-09-20T19:38:46.754 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi114') 2024-09-20T19:38:46.754 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi114') 2024-09-20T19:38:46.755 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi114') 2024-09-20T19:38:46.755 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi114') 2024-09-20T19:38:47.389 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:47 smithi114 ceph-mon[28260]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:47.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.586+0000 7f7cd6972640 1 -- 172.21.15.114:0/3603884189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cd0101a40 msgr2=0x7f7cd0101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:38:47.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.586+0000 7f7cd6972640 1 --2- 172.21.15.114:0/3603884189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cd0101a40 0x7f7cd0101ec0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009880 tx=0x7f7cc402a4a0 comp rx=0 tx=0).stop 2024-09-20T19:38:47.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.586+0000 7f7cd6972640 1 -- 172.21.15.114:0/3603884189 shutdown_connections 2024-09-20T19:38:47.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.586+0000 7f7cd6972640 1 --2- 172.21.15.114:0/3603884189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cd0101a40 0x7f7cd0101ec0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:47.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.586+0000 7f7cd6972640 1 --2- 172.21.15.114:0/3603884189 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cd0100840 0x7f7cd0100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:47.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.586+0000 7f7cd6972640 1 -- 172.21.15.114:0/3603884189 >> 172.21.15.114:0/3603884189 conn(0x7f7cd00fbff0 msgr2=0x7f7cd00fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:38:47.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.586+0000 7f7cd6972640 1 -- 172.21.15.114:0/3603884189 shutdown_connections 2024-09-20T19:38:47.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.586+0000 7f7cd6972640 1 -- 172.21.15.114:0/3603884189 wait complete. 2024-09-20T19:38:47.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.587+0000 7f7cd6972640 1 Processor -- start 2024-09-20T19:38:47.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.587+0000 7f7cd6972640 1 -- start start 2024-09-20T19:38:47.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.587+0000 7f7cd6972640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cd0100840 0x7f7cd01049f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:38:47.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.587+0000 7f7cd6972640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cd0101a40 0x7f7cd01030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:38:47.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.587+0000 7f7cd6972640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd0109700 con 0x7f7cd0100840 2024-09-20T19:38:47.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.587+0000 7f7cd6972640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd0109580 con 0x7f7cd0101a40 2024-09-20T19:38:47.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.587+0000 7f7ccffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cd0100840 0x7f7cd01049f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:38:47.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.587+0000 7f7ccffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cd0100840 0x7f7cd01049f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44912/0 (socket says 172.21.15.114:44912) 2024-09-20T19:38:47.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.587+0000 7f7ccffff640 1 -- 172.21.15.114:0/963900602 learned_addr learned my addr 172.21.15.114:0/963900602 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:38:47.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.588+0000 7f7ccffff640 1 -- 172.21.15.114:0/963900602 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cd0101a40 msgr2=0x7f7cd01030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:38:47.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.588+0000 7f7ccffff640 1 --2- 172.21.15.114:0/963900602 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cd0101a40 0x7f7cd01030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:47.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.588+0000 7f7ccffff640 1 -- 172.21.15.114:0/963900602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd01035f0 con 0x7f7cd0100840 2024-09-20T19:38:47.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.588+0000 7f7ccffff640 1 --2- 172.21.15.114:0/963900602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cd0100840 0x7f7cd01049f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000b3d0 tx=0x7f7cc000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:38:47.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.588+0000 7f7ccd7fa640 1 -- 172.21.15.114:0/963900602 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f7cc0015020 con 0x7f7cd0100840 2024-09-20T19:38:47.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.588+0000 7f7ccd7fa640 1 -- 172.21.15.114:0/963900602 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7cc0011af0 con 0x7f7cd0100840 2024-09-20T19:38:47.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.588+0000 7f7ccd7fa640 1 -- 172.21.15.114:0/963900602 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f7cc0010da0 con 0x7f7cd0100840 2024-09-20T19:38:47.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.589+0000 7f7cd6972640 1 -- 172.21.15.114:0/963900602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cd01038e0 con 0x7f7cd0100840 2024-09-20T19:38:47.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.589+0000 7f7cd6972640 1 -- 172.21.15.114:0/963900602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cd0103ef0 con 0x7f7cd0100840 2024-09-20T19:38:47.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.590+0000 7f7cd6972640 1 -- 172.21.15.114:0/963900602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cd0100c40 con 0x7f7cd0100840 2024-09-20T19:38:47.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.590+0000 7f7ccd7fa640 1 -- 172.21.15.114:0/963900602 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 92497+0+0 (secure 0 0 0) 0x7f7cc0018020 con 0x7f7cd0100840 2024-09-20T19:38:47.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.591+0000 7f7ccd7fa640 1 --2- 172.21.15.114:0/963900602 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7cb806e230 0x7f7cb80706f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:38:47.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.591+0000 7f7ccd7fa640 1 -- 172.21.15.114:0/963900602 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f7cc008cbe0 con 0x7f7cd0100840 2024-09-20T19:38:47.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.591+0000 7f7ccf7fe640 1 --2- 172.21.15.114:0/963900602 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7cb806e230 0x7f7cb80706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:38:47.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.591+0000 7f7ccf7fe640 1 --2- 172.21.15.114:0/963900602 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7cb806e230 0x7f7cb80706f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7cc40096c0 tx=0x7f7cc4000bc0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:38:47.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.595+0000 7f7ccd7fa640 1 -- 172.21.15.114:0/963900602 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f7cc001f020 con 0x7f7cd0100840 2024-09-20T19:38:47.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.771+0000 7f7cd6972640 1 -- 172.21.15.114:0/963900602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7f7cd0106850 con 0x7f7cd0100840 2024-09-20T19:38:47.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.774+0000 7f7ccd7fa640 1 -- 172.21.15.114:0/963900602 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v27)=0 v27) ==== 119+0+0 (secure 0 0 0) 0x7f7cc005b5c0 con 0x7f7cd0100840 2024-09-20T19:38:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.776+0000 7f7cd6972640 1 -- 172.21.15.114:0/963900602 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7cb806e230 msgr2=0x7f7cb80706f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:38:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.776+0000 7f7cd6972640 1 --2- 172.21.15.114:0/963900602 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7cb806e230 0x7f7cb80706f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7cc40096c0 tx=0x7f7cc4000bc0 comp rx=0 tx=0).stop 2024-09-20T19:38:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.776+0000 7f7cd6972640 1 -- 172.21.15.114:0/963900602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cd0100840 msgr2=0x7f7cd01049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:38:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.776+0000 7f7cd6972640 1 --2- 172.21.15.114:0/963900602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cd0100840 0x7f7cd01049f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000b3d0 tx=0x7f7cc000b890 comp rx=0 tx=0).stop 2024-09-20T19:38:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.777+0000 7f7cd6972640 1 -- 172.21.15.114:0/963900602 shutdown_connections 2024-09-20T19:38:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.777+0000 7f7cd6972640 1 --2- 172.21.15.114:0/963900602 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7cb806e230 0x7f7cb80706f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.777+0000 7f7cd6972640 1 --2- 172.21.15.114:0/963900602 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cd0101a40 0x7f7cd01030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:47.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.777+0000 7f7cd6972640 1 --2- 172.21.15.114:0/963900602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cd0100840 0x7f7cd01049f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.777+0000 7f7cd6972640 1 -- 172.21.15.114:0/963900602 >> 172.21.15.114:0/963900602 conn(0x7f7cd00fbff0 msgr2=0x7f7cd00fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:38:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.777+0000 7f7cd6972640 1 -- 172.21.15.114:0/963900602 shutdown_connections 2024-09-20T19:38:47.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:47.777+0000 7f7cd6972640 1 -- 172.21.15.114:0/963900602 wait complete. 2024-09-20T19:38:47.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:47 smithi160 ceph-mon[35859]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:48.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.567+0000 7f8dff405640 1 -- 172.21.15.114:0/2444336511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df8102410 msgr2=0x7f8df8104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:38:48.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.567+0000 7f8dff405640 1 --2- 172.21.15.114:0/2444336511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df8102410 0x7f8df8104800 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8df40098b0 tx=0x7f8df402a4d0 comp rx=0 tx=0).stop 2024-09-20T19:38:48.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.567+0000 7f8dff405640 1 -- 172.21.15.114:0/2444336511 shutdown_connections 2024-09-20T19:38:48.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.567+0000 7f8dff405640 1 --2- 172.21.15.114:0/2444336511 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8df8104d40 0x7f8df8107130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:48.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.567+0000 7f8dff405640 1 --2- 172.21.15.114:0/2444336511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df8102410 0x7f8df8104800 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:48.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.567+0000 7f8dff405640 1 -- 172.21.15.114:0/2444336511 >> 172.21.15.114:0/2444336511 conn(0x7f8df80fc010 msgr2=0x7f8df80fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:38:48.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.567+0000 7f8dff405640 1 -- 172.21.15.114:0/2444336511 shutdown_connections 2024-09-20T19:38:48.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.567+0000 7f8dff405640 1 -- 172.21.15.114:0/2444336511 wait complete. 2024-09-20T19:38:48.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.568+0000 7f8dff405640 1 Processor -- start 2024-09-20T19:38:48.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.568+0000 7f8dff405640 1 -- start start 2024-09-20T19:38:48.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.568+0000 7f8dff405640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8df8102410 0x7f8df8197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:38:48.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.568+0000 7f8dff405640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df8104d40 0x7f8df8197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:38:48.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.568+0000 7f8dff405640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8df810a850 con 0x7f8df8104d40 2024-09-20T19:38:48.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.568+0000 7f8dff405640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8df810a6d0 con 0x7f8df8102410 2024-09-20T19:38:48.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.568+0000 7f8dfc9a0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df8104d40 0x7f8df8197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:38:48.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.568+0000 7f8dfc9a0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df8104d40 0x7f8df8197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44930/0 (socket says 172.21.15.114:44930) 2024-09-20T19:38:48.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.568+0000 7f8dfc9a0640 1 -- 172.21.15.114:0/840063743 learned_addr learned my addr 172.21.15.114:0/840063743 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:38:48.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.568+0000 7f8dfd1a1640 1 --2- 172.21.15.114:0/840063743 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8df8102410 0x7f8df8197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:38:48.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.568+0000 7f8dfc9a0640 1 -- 172.21.15.114:0/840063743 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8df8102410 msgr2=0x7f8df8197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:38:48.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.568+0000 7f8dfc9a0640 1 --2- 172.21.15.114:0/840063743 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8df8102410 0x7f8df8197570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:48.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.568+0000 7f8dfc9a0640 1 -- 172.21.15.114:0/840063743 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df8198130 con 0x7f8df8104d40 2024-09-20T19:38:48.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.568+0000 7f8dfc9a0640 1 --2- 172.21.15.114:0/840063743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df8104d40 0x7f8df8197ab0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8de800b3d0 tx=0x7f8de800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:38:48.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.568+0000 7f8dee7fc640 1 -- 172.21.15.114:0/840063743 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8de8015020 con 0x7f8df8104d40 2024-09-20T19:38:48.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.569+0000 7f8dff405640 1 -- 172.21.15.114:0/840063743 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df819cb70 con 0x7f8df8104d40 2024-09-20T19:38:48.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.569+0000 7f8dee7fc640 1 -- 172.21.15.114:0/840063743 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8de8011af0 con 0x7f8df8104d40 2024-09-20T19:38:48.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.569+0000 7f8dee7fc640 1 -- 172.21.15.114:0/840063743 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8de8010d30 con 0x7f8df8104d40 2024-09-20T19:38:48.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.569+0000 7f8dff405640 1 -- 172.21.15.114:0/840063743 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df819d180 con 0x7f8df8104d40 2024-09-20T19:38:48.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.570+0000 7f8dff405640 1 -- 172.21.15.114:0/840063743 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8df81027f0 con 0x7f8df8104d40 2024-09-20T19:38:48.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.570+0000 7f8dee7fc640 1 -- 172.21.15.114:0/840063743 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 92497+0+0 (secure 0 0 0) 0x7f8de8018020 con 0x7f8df8104d40 2024-09-20T19:38:48.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.571+0000 7f8dee7fc640 1 --2- 172.21.15.114:0/840063743 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8de406e230 0x7f8de40706f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:38:48.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.571+0000 7f8dfd1a1640 1 --2- 172.21.15.114:0/840063743 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8de406e230 0x7f8de40706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:38:48.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.571+0000 7f8dee7fc640 1 -- 172.21.15.114:0/840063743 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8de808cb30 con 0x7f8df8104d40 2024-09-20T19:38:48.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.571+0000 7f8dfd1a1640 1 --2- 172.21.15.114:0/840063743 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8de406e230 0x7f8de40706f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8df4000c00 tx=0x7f8df4005a20 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:38:48.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.575+0000 7f8dee7fc640 1 -- 172.21.15.114:0/840063743 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f8de801f020 con 0x7f8df8104d40 2024-09-20T19:38:48.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.749+0000 7f8dff405640 1 -- 172.21.15.114:0/840063743 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f8df819d410 con 0x7f8df8104d40 2024-09-20T19:38:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.752+0000 7f8dee7fc640 1 -- 172.21.15.114:0/840063743 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v28)=0 v28) ==== 106+0+0 (secure 0 0 0) 0x7f8de805b570 con 0x7f8df8104d40 2024-09-20T19:38:48.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.755+0000 7f8dff405640 1 -- 172.21.15.114:0/840063743 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8de406e230 msgr2=0x7f8de40706f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:38:48.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.755+0000 7f8dff405640 1 --2- 172.21.15.114:0/840063743 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8de406e230 0x7f8de40706f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8df4000c00 tx=0x7f8df4005a20 comp rx=0 tx=0).stop 2024-09-20T19:38:48.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.755+0000 7f8dff405640 1 -- 172.21.15.114:0/840063743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df8104d40 msgr2=0x7f8df8197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:38:48.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.755+0000 7f8dff405640 1 --2- 172.21.15.114:0/840063743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df8104d40 0x7f8df8197ab0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8de800b3d0 tx=0x7f8de800b890 comp rx=0 tx=0).stop 2024-09-20T19:38:48.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.755+0000 7f8dff405640 1 -- 172.21.15.114:0/840063743 shutdown_connections 2024-09-20T19:38:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.755+0000 7f8dff405640 1 --2- 172.21.15.114:0/840063743 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8de406e230 0x7f8de40706f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.755+0000 7f8dff405640 1 --2- 172.21.15.114:0/840063743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df8104d40 0x7f8df8197ab0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.755+0000 7f8dff405640 1 --2- 172.21.15.114:0/840063743 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8df8102410 0x7f8df8197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.755+0000 7f8dff405640 1 -- 172.21.15.114:0/840063743 >> 172.21.15.114:0/840063743 conn(0x7f8df80fc010 msgr2=0x7f8df80fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:38:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.755+0000 7f8dff405640 1 -- 172.21.15.114:0/840063743 shutdown_connections 2024-09-20T19:38:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:48.755+0000 7f8dff405640 1 -- 172.21.15.114:0/840063743 wait complete. 2024-09-20T19:38:48.995 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:48 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/963900602' entity='client.admin' 2024-09-20T19:38:48.995 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:48 smithi114 ceph-mon[28260]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:48.995 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:48 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/840063743' entity='client.admin' 2024-09-20T19:38:49.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:48 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/963900602' entity='client.admin' 2024-09-20T19:38:49.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:48 smithi160 ceph-mon[35859]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:49.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:48 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/840063743' entity='client.admin' 2024-09-20T19:38:49.611 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi114') 2024-09-20T19:38:49.611 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi114') 2024-09-20T19:38:49.611 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi114') 2024-09-20T19:38:49.612 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi114') 2024-09-20T19:38:49.612 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi114') 2024-09-20T19:38:49.612 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi114') 2024-09-20T19:38:49.612 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi114') 2024-09-20T19:38:49.612 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi114') 2024-09-20T19:38:49.612 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi114') 2024-09-20T19:38:49.612 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '6a49c7bc-7787-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi114') 2024-09-20T19:38:49.612 INFO:tasks.cephadm:Deploying OSDs... 2024-09-20T19:38:49.613 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:38:49.613 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T19:38:49.631 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T19:38:49.631 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-09-20T19:38:49.688 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T19:38:49.688 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:38:49.688 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-20T19:38:49.688 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:38:49.688 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:38:49.688 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-20 19:37:49.799968708 +0000 2024-09-20T19:38:49.689 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-20 19:32:27.629291340 +0000 2024-09-20T19:38:49.689 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-20 19:32:27.629291340 +0000 2024-09-20T19:38:49.689 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-20 19:32:27.629291340 +0000 2024-09-20T19:38:49.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T19:38:49.755 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-20T19:38:49.755 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-20T19:38:49.756 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000216146 s, 2.4 MB/s 2024-09-20T19:38:49.757 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T19:38:49.814 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-09-20T19:38:49.871 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T19:38:49.871 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:38:49.871 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-20T19:38:49.871 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:38:49.871 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:38:49.871 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-20 19:37:49.799968708 +0000 2024-09-20T19:38:49.871 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-20 19:32:27.944284242 +0000 2024-09-20T19:38:49.871 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-20 19:32:27.944284242 +0000 2024-09-20T19:38:49.872 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-20 19:32:27.944284242 +0000 2024-09-20T19:38:49.872 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T19:38:49.935 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-20T19:38:49.935 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-20T19:38:49.936 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000173809 s, 2.9 MB/s 2024-09-20T19:38:49.938 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T19:38:49.995 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-09-20T19:38:50.054 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T19:38:50.054 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:38:50.054 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-20T19:38:50.054 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:38:50.054 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:38:50.054 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-20 19:37:49.800968686 +0000 2024-09-20T19:38:50.054 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-20 19:32:28.274276805 +0000 2024-09-20T19:38:50.054 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-20 19:32:28.274276805 +0000 2024-09-20T19:38:50.055 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-20 19:32:28.274276805 +0000 2024-09-20T19:38:50.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T19:38:50.120 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-20T19:38:50.121 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-20T19:38:50.121 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000171842 s, 3.0 MB/s 2024-09-20T19:38:50.123 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T19:38:50.179 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-09-20T19:38:50.235 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T19:38:50.235 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:38:50.235 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-20T19:38:50.235 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:38:50.235 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:38:50.235 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-20 19:37:49.800968686 +0000 2024-09-20T19:38:50.235 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-20 19:32:28.607269302 +0000 2024-09-20T19:38:50.235 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-20 19:32:28.607269302 +0000 2024-09-20T19:38:50.235 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-20 19:32:28.607269302 +0000 2024-09-20T19:38:50.236 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T19:38:50.299 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-20T19:38:50.299 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-20T19:38:50.299 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000221216 s, 2.3 MB/s 2024-09-20T19:38:50.300 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T19:38:50.356 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-20T19:38:50.356 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T19:38:50.377 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T19:38:50.377 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_1 2024-09-20T19:38:50.436 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T19:38:50.436 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:38:50.436 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-20T19:38:50.436 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:38:50.436 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:38:50.436 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-20 19:38:19.470472486 +0000 2024-09-20T19:38:50.436 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-20 19:32:27.624717466 +0000 2024-09-20T19:38:50.436 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-20 19:32:27.624717466 +0000 2024-09-20T19:38:50.436 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-20 19:32:27.624717466 +0000 2024-09-20T19:38:50.437 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T19:38:50.506 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-20T19:38:50.507 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-20T19:38:50.507 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000248945 s, 2.1 MB/s 2024-09-20T19:38:50.508 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T19:38:50.568 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_2 2024-09-20T19:38:50.627 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T19:38:50.627 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:38:50.627 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-20T19:38:50.627 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:38:50.628 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:38:50.628 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-20 19:38:19.471472472 +0000 2024-09-20T19:38:50.628 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-20 19:32:27.944713220 +0000 2024-09-20T19:38:50.628 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-20 19:32:27.944713220 +0000 2024-09-20T19:38:50.628 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-20 19:32:27.944713220 +0000 2024-09-20T19:38:50.629 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T19:38:50.695 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-20T19:38:50.695 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-20T19:38:50.696 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000231791 s, 2.2 MB/s 2024-09-20T19:38:50.697 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T19:38:50.757 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_3 2024-09-20T19:38:50.816 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T19:38:50.816 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:38:50.816 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-20T19:38:50.816 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:38:50.816 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:38:50.816 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-20 19:38:19.471472472 +0000 2024-09-20T19:38:50.816 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-20 19:32:28.252709135 +0000 2024-09-20T19:38:50.817 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-20 19:32:28.252709135 +0000 2024-09-20T19:38:50.817 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-20 19:32:28.252709135 +0000 2024-09-20T19:38:50.817 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T19:38:50.883 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-20T19:38:50.883 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-20T19:38:50.883 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000246677 s, 2.1 MB/s 2024-09-20T19:38:50.884 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T19:38:50.943 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_4 2024-09-20T19:38:51.001 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T19:38:51.001 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T19:38:51.001 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-20T19:38:51.001 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:38:51.001 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T19:38:51.001 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-20 19:38:19.471472472 +0000 2024-09-20T19:38:51.001 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-20 19:32:28.578704811 +0000 2024-09-20T19:38:51.001 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-20 19:32:28.578704811 +0000 2024-09-20T19:38:51.001 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-20 19:32:28.578704811 +0000 2024-09-20T19:38:51.002 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T19:38:51.067 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-20T19:38:51.067 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-20T19:38:51.067 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000252756 s, 2.0 MB/s 2024-09-20T19:38:51.068 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T19:38:51.125 INFO:tasks.cephadm:Deploying osd.0 on smithi114 with /dev/vg_nvme/lv_4... 2024-09-20T19:38:51.125 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T19:38:51.425 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:51 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:51.425 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:51 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:51.425 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:51 smithi114 ceph-mon[28260]: Reconfiguring ceph-exporter.smithi114 (monmap changed)... 2024-09-20T19:38:51.425 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:51 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi114", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-20T19:38:51.425 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:51 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:51.425 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:51 smithi114 ceph-mon[28260]: Reconfiguring daemon ceph-exporter.smithi114 on smithi114 2024-09-20T19:38:51.425 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:51 smithi114 ceph-mon[28260]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:51.504 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:51.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:51 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:51.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:51 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:51.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:51 smithi160 ceph-mon[35859]: Reconfiguring ceph-exporter.smithi114 (monmap changed)... 2024-09-20T19:38:51.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:51 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi114", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-20T19:38:51.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:51 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:51.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:51 smithi160 ceph-mon[35859]: Reconfiguring daemon ceph-exporter.smithi114 on smithi114 2024-09-20T19:38:51.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:51 smithi160 ceph-mon[35859]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:52.572 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:52 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:52.572 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:52 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:52.573 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:52 smithi114 ceph-mon[28260]: Reconfiguring crash.smithi114 (monmap changed)... 2024-09-20T19:38:52.573 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:52 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi114", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T19:38:52.573 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:52 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:52.573 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:52 smithi114 ceph-mon[28260]: Reconfiguring daemon crash.smithi114 on smithi114 2024-09-20T19:38:52.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:52 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:52.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:52 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:52.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:52 smithi160 ceph-mon[35859]: Reconfiguring crash.smithi114 (monmap changed)... 2024-09-20T19:38:52.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:52 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi114", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T19:38:52.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:52 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:52.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:52 smithi160 ceph-mon[35859]: Reconfiguring daemon crash.smithi114 on smithi114 2024-09-20T19:38:53.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:53 smithi160 ceph-mon[35859]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:53.889 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:53 smithi114 ceph-mon[28260]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:55.003 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:54 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:55.003 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:54 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:55.003 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:54 smithi114 ceph-mon[28260]: Reconfiguring alertmanager.smithi114 (dependencies changed)... 2024-09-20T19:38:55.003 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:54 smithi114 ceph-mon[28260]: Reconfiguring daemon alertmanager.smithi114 on smithi114 2024-09-20T19:38:55.003 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:54 smithi114 ceph-mon[28260]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:55.006 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:38:55.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:54 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:55.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:54 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:38:55.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:54 smithi160 ceph-mon[35859]: Reconfiguring alertmanager.smithi114 (dependencies changed)... 2024-09-20T19:38:55.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:54 smithi160 ceph-mon[35859]: Reconfiguring daemon alertmanager.smithi114 on smithi114 2024-09-20T19:38:55.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:54 smithi160 ceph-mon[35859]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:55.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi114:vg_nvme/lv_4 2024-09-20T19:38:55.833 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:38:56.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.234+0000 7ff7c6c4f640 1 -- 172.21.15.114:0/825788527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7c0100880 msgr2=0x7ff7c0100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:38:56.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.234+0000 7ff7c6c4f640 1 --2- 172.21.15.114:0/825788527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7c0100880 0x7ff7c0100c80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009880 tx=0x7ff7b402a4a0 comp rx=0 tx=0).stop 2024-09-20T19:38:56.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.234+0000 7ff7c6c4f640 1 -- 172.21.15.114:0/825788527 shutdown_connections 2024-09-20T19:38:56.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.234+0000 7ff7c6c4f640 1 --2- 172.21.15.114:0/825788527 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff7c0101a80 0x7ff7c0101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:56.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.234+0000 7ff7c6c4f640 1 --2- 172.21.15.114:0/825788527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7c0100880 0x7ff7c0100c80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.234+0000 7ff7c6c4f640 1 -- 172.21.15.114:0/825788527 >> 172.21.15.114:0/825788527 conn(0x7ff7c00fc010 msgr2=0x7ff7c00fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:38:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.234+0000 7ff7c6c4f640 1 -- 172.21.15.114:0/825788527 shutdown_connections 2024-09-20T19:38:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.234+0000 7ff7c6c4f640 1 -- 172.21.15.114:0/825788527 wait complete. 2024-09-20T19:38:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.235+0000 7ff7c6c4f640 1 Processor -- start 2024-09-20T19:38:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.235+0000 7ff7c6c4f640 1 -- start start 2024-09-20T19:38:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.235+0000 7ff7c6c4f640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff7c0100880 0x7ff7c006adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:38:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.235+0000 7ff7c6c4f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7c0101a80 0x7ff7c019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:38:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.235+0000 7ff7c6c4f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c0109720 con 0x7ff7c0101a80 2024-09-20T19:38:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.235+0000 7ff7c6c4f640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c01095a0 con 0x7ff7c0100880 2024-09-20T19:38:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.235+0000 7ff7bffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7c0101a80 0x7ff7c019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:38:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.235+0000 7ff7bffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7c0101a80 0x7ff7c019e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38500/0 (socket says 172.21.15.114:38500) 2024-09-20T19:38:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.235+0000 7ff7bffff640 1 -- 172.21.15.114:0/1448943860 learned_addr learned my addr 172.21.15.114:0/1448943860 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:38:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.235+0000 7ff7bffff640 1 -- 172.21.15.114:0/1448943860 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff7c0100880 msgr2=0x7ff7c006adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:38:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.235+0000 7ff7bffff640 1 --2- 172.21.15.114:0/1448943860 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff7c0100880 0x7ff7c006adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:38:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.235+0000 7ff7bffff640 1 -- 172.21.15.114:0/1448943860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c019ebb0 con 0x7ff7c0101a80 2024-09-20T19:38:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.236+0000 7ff7bffff640 1 --2- 172.21.15.114:0/1448943860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7c0101a80 0x7ff7c019e670 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00b420 tx=0x7ff7ac00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:38:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.236+0000 7ff7bdffb640 1 -- 172.21.15.114:0/1448943860 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff7ac011630 con 0x7ff7c0101a80 2024-09-20T19:38:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.236+0000 7ff7bdffb640 1 -- 172.21.15.114:0/1448943860 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff7ac016070 con 0x7ff7c0101a80 2024-09-20T19:38:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.236+0000 7ff7c6c4f640 1 -- 172.21.15.114:0/1448943860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7c019ed80 con 0x7ff7c0101a80 2024-09-20T19:38:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.236+0000 7ff7bdffb640 1 -- 172.21.15.114:0/1448943860 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff7ac010d30 con 0x7ff7c0101a80 2024-09-20T19:38:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.236+0000 7ff7c6c4f640 1 -- 172.21.15.114:0/1448943860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7c019f330 con 0x7ff7c0101a80 2024-09-20T19:38:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.237+0000 7ff7c6c4f640 1 -- 172.21.15.114:0/1448943860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7c0100c80 con 0x7ff7c0101a80 2024-09-20T19:38:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.238+0000 7ff7bdffb640 1 -- 172.21.15.114:0/1448943860 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 92497+0+0 (secure 0 0 0) 0x7ff7ac01b480 con 0x7ff7c0101a80 2024-09-20T19:38:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.238+0000 7ff7bdffb640 1 --2- 172.21.15.114:0/1448943860 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff7b006e230 0x7ff7b00706f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:38:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.238+0000 7ff7bdffb640 1 -- 172.21.15.114:0/1448943860 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7ff7ac08c8e0 con 0x7ff7c0101a80 2024-09-20T19:38:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.238+0000 7ff7c49eb640 1 --2- 172.21.15.114:0/1448943860 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff7b006e230 0x7ff7b00706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:38:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.239+0000 7ff7c49eb640 1 --2- 172.21.15.114:0/1448943860 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff7b006e230 0x7ff7b00706f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff7b40096c0 tx=0x7ff7b4000bc0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:38:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.243+0000 7ff7bdffb640 1 -- 172.21.15.114:0/1448943860 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7ff7ac05b290 con 0x7ff7c0101a80 2024-09-20T19:38:56.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:38:56.428+0000 7ff7c6c4f640 1 -- 172.21.15.114:0/1448943860 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ff7c0106000 con 0x7ff7b006e230 2024-09-20T19:38:56.646 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:56 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:38:56.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:56 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:38:57.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:57 smithi114 ceph-mon[28260]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:57.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:57 smithi114 ceph-mon[28260]: from='client.14330 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:38:57.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:57 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:38:57.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:57 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:38:57.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:57 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:57.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:57 smithi160 ceph-mon[35859]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:57.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:57 smithi160 ceph-mon[35859]: from='client.14330 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:38:57.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:57 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:38:57.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:57 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:38:57.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:57 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:38:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:38:59 smithi114 ceph-mon[28260]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:38:59.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:38:59 smithi160 ceph-mon[35859]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:01.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:01 smithi114 ceph-mon[28260]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:01.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:01 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:01.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:01 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:01.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:01 smithi160 ceph-mon[35859]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:01.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:01 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:01.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:01 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:02.696 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:02 smithi114 ceph-mon[28260]: Reconfiguring grafana.smithi114 (dependencies changed)... 2024-09-20T19:39:02.696 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:02 smithi114 ceph-mon[28260]: Reconfiguring daemon grafana.smithi114 on smithi114 2024-09-20T19:39:02.696 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:02 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/348071859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ac4e0d5-030f-4de4-96a7-fd80f6c10a78"}]: dispatch 2024-09-20T19:39:02.696 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:02 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/348071859' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ac4e0d5-030f-4de4-96a7-fd80f6c10a78"}]': finished 2024-09-20T19:39:02.697 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:02 smithi114 ceph-mon[28260]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T19:39:02.697 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:02 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:39:02.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:02 smithi160 ceph-mon[35859]: Reconfiguring grafana.smithi114 (dependencies changed)... 2024-09-20T19:39:02.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:02 smithi160 ceph-mon[35859]: Reconfiguring daemon grafana.smithi114 on smithi114 2024-09-20T19:39:02.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:02 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/348071859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ac4e0d5-030f-4de4-96a7-fd80f6c10a78"}]: dispatch 2024-09-20T19:39:02.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:02 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/348071859' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ac4e0d5-030f-4de4-96a7-fd80f6c10a78"}]': finished 2024-09-20T19:39:02.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:02 smithi160 ceph-mon[35859]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T19:39:02.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:02 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:39:03.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:03 smithi114 ceph-mon[28260]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:03.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:03 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3608234543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:39:03.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:03 smithi160 ceph-mon[35859]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:03.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:03 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/3608234543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:39:05.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:05 smithi114 ceph-mon[28260]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:05.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:05 smithi160 ceph-mon[35859]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:07.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:07 smithi114 ceph-mon[28260]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:07.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:07 smithi160 ceph-mon[35859]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:09.647 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:09 smithi114 ceph-mon[28260]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:09.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:09 smithi160 ceph-mon[35859]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:11.627 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:11 smithi114 ceph-mon[28260]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:11.627 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:11 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:39:11.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:11 smithi160 ceph-mon[35859]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:11.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:11 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:39:13.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:13 smithi114 ceph-mon[28260]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:13.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:13 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:13.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:13 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:13.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:13 smithi160 ceph-mon[35859]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:13.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:13 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:13.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:13 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:14.639 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:14 smithi114 ceph-mon[28260]: Reconfiguring prometheus.smithi114 (dependencies changed)... 2024-09-20T19:39:14.639 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:14 smithi114 ceph-mon[28260]: Reconfiguring daemon prometheus.smithi114 on smithi114 2024-09-20T19:39:14.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:14 smithi160 ceph-mon[35859]: Reconfiguring prometheus.smithi114 (dependencies changed)... 2024-09-20T19:39:14.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:14 smithi160 ceph-mon[35859]: Reconfiguring daemon prometheus.smithi114 on smithi114 2024-09-20T19:39:15.637 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:15 smithi114 ceph-mon[28260]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:15.637 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:15 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T19:39:15.637 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:15 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:15.637 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:15 smithi114 ceph-mon[28260]: Deploying daemon osd.0 on smithi114 2024-09-20T19:39:15.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:15 smithi160 ceph-mon[35859]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:15.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:15 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T19:39:15.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:15 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:15.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:15 smithi160 ceph-mon[35859]: Deploying daemon osd.0 on smithi114 2024-09-20T19:39:17.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:17 smithi114 ceph-mon[28260]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:17.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:17 smithi160 ceph-mon[35859]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:19.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:19 smithi114 ceph-mon[28260]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:19.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:19 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:19.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:19 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:19.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:19 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi160", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-20T19:39:19.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:19 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:19.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:19 smithi160 ceph-mon[35859]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:19.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:19 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:19.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:19 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:19.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:19 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi160", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-20T19:39:19.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:19 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:20.409 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:20 smithi114 ceph-mon[28260]: Reconfiguring ceph-exporter.smithi160 (monmap changed)... 2024-09-20T19:39:20.409 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:20 smithi114 ceph-mon[28260]: Reconfiguring daemon ceph-exporter.smithi160 on smithi160 2024-09-20T19:39:20.409 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:20 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:20.409 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:20 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:20.409 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:20 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi160", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T19:39:20.409 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:20 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:20.600 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:20 smithi160 ceph-mon[35859]: Reconfiguring ceph-exporter.smithi160 (monmap changed)... 2024-09-20T19:39:20.600 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:20 smithi160 ceph-mon[35859]: Reconfiguring daemon ceph-exporter.smithi160 on smithi160 2024-09-20T19:39:20.600 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:20.601 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:20.601 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi160", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T19:39:20.601 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:20 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:21.704 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:21 smithi160 ceph-mon[35859]: Reconfiguring crash.smithi160 (monmap changed)... 2024-09-20T19:39:21.705 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:21 smithi160 ceph-mon[35859]: Reconfiguring daemon crash.smithi160 on smithi160 2024-09-20T19:39:21.705 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:21 smithi160 ceph-mon[35859]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:21.705 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:21 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:21.705 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:21 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:21.705 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:21 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi160.bspiak", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:39:21.705 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:21 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:39:21.705 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:21 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:21.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:21 smithi114 ceph-mon[28260]: Reconfiguring crash.smithi160 (monmap changed)... 2024-09-20T19:39:21.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:21 smithi114 ceph-mon[28260]: Reconfiguring daemon crash.smithi160 on smithi160 2024-09-20T19:39:21.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:21 smithi114 ceph-mon[28260]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:21.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:21 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:21.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:21 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:21.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:21 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi160.bspiak", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T19:39:21.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:21 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T19:39:21.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:21 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:22.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:22 smithi114 ceph-mon[28260]: Reconfiguring mgr.smithi160.bspiak (monmap changed)... 2024-09-20T19:39:22.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:22 smithi114 ceph-mon[28260]: Reconfiguring daemon mgr.smithi160.bspiak on smithi160 2024-09-20T19:39:22.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:22 smithi160 ceph-mon[35859]: Reconfiguring mgr.smithi160.bspiak (monmap changed)... 2024-09-20T19:39:22.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:22 smithi160 ceph-mon[35859]: Reconfiguring daemon mgr.smithi160.bspiak on smithi160 2024-09-20T19:39:23.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:23 smithi160 ceph-mon[35859]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:23.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:23 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:23.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:23 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:23.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:23 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:39:23.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:23 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:39:23.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:23 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:23.601 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:23 smithi114 ceph-mon[28260]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:23.602 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:23 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:23.602 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:23 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:23.602 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:23 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T19:39:23.602 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:23 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T19:39:23.602 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:23 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:24.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:24 smithi114 ceph-mon[28260]: Reconfiguring mon.smithi160 (monmap changed)... 2024-09-20T19:39:24.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:24 smithi114 ceph-mon[28260]: Reconfiguring daemon mon.smithi160 on smithi160 2024-09-20T19:39:24.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:24.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:24.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:39:24.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:39:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:39:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:39:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:39:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:39:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:39:24.665 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:24 smithi160 ceph-mon[35859]: Reconfiguring mon.smithi160 (monmap changed)... 2024-09-20T19:39:24.665 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:24 smithi160 ceph-mon[35859]: Reconfiguring daemon mon.smithi160 on smithi160 2024-09-20T19:39:24.665 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:24.665 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:24.665 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:39:24.666 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:39:24.666 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:24.666 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:39:24.666 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:39:24.666 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:24.666 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:39:24.666 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:39:24.666 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:24.666 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:39:25.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:25 smithi114 ceph-mon[28260]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:39:25.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:25 smithi114 ceph-mon[28260]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:39:25.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:25 smithi114 ceph-mon[28260]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:39:25.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:25 smithi114 ceph-mon[28260]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:39:25.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:25 smithi114 ceph-mon[28260]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:39:25.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:25 smithi114 ceph-mon[28260]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:39:25.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:25 smithi114 ceph-mon[28260]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:25.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:25.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:25.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:25 smithi160 ceph-mon[35859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T19:39:25.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:25 smithi160 ceph-mon[35859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T19:39:25.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:25 smithi160 ceph-mon[35859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T19:39:25.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:25 smithi160 ceph-mon[35859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T19:39:25.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:25 smithi160 ceph-mon[35859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T19:39:25.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:25 smithi160 ceph-mon[35859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T19:39:25.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:25 smithi160 ceph-mon[35859]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:25.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:25.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:26.772 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:26 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:26.772 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:26 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:26.772 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:26 smithi114 ceph-mon[28260]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:26.772 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:26 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:39:27.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:26 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:27.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:26 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:27.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:26 smithi160 ceph-mon[35859]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:27.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:26 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:39:28.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:28.941+0000 7ff7bdffb640 1 -- 172.21.15.114:0/1448943860 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff7c0106000 con 0x7ff7b006e230 2024-09-20T19:39:28.943 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 0 on host 'smithi114' 2024-09-20T19:39:28.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:28.944+0000 7ff7c6c4f640 1 -- 172.21.15.114:0/1448943860 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff7b006e230 msgr2=0x7ff7b00706f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:39:28.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:28.944+0000 7ff7c6c4f640 1 --2- 172.21.15.114:0/1448943860 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff7b006e230 0x7ff7b00706f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff7b40096c0 tx=0x7ff7b4000bc0 comp rx=0 tx=0).stop 2024-09-20T19:39:28.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:28.944+0000 7ff7c6c4f640 1 -- 172.21.15.114:0/1448943860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7c0101a80 msgr2=0x7ff7c019e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:39:28.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:28.944+0000 7ff7c6c4f640 1 --2- 172.21.15.114:0/1448943860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7c0101a80 0x7ff7c019e670 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00b420 tx=0x7ff7ac00b8e0 comp rx=0 tx=0).stop 2024-09-20T19:39:28.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:28.944+0000 7ff7c6c4f640 1 -- 172.21.15.114:0/1448943860 shutdown_connections 2024-09-20T19:39:28.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:28.944+0000 7ff7c6c4f640 1 --2- 172.21.15.114:0/1448943860 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff7b006e230 0x7ff7b00706f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:39:28.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:28.944+0000 7ff7c6c4f640 1 --2- 172.21.15.114:0/1448943860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7c0101a80 0x7ff7c019e670 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:39:28.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:28.944+0000 7ff7c6c4f640 1 --2- 172.21.15.114:0/1448943860 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff7c0100880 0x7ff7c006adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:39:28.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:28.944+0000 7ff7c6c4f640 1 -- 172.21.15.114:0/1448943860 >> 172.21.15.114:0/1448943860 conn(0x7ff7c00fc010 msgr2=0x7ff7c00fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:39:28.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:28.944+0000 7ff7c6c4f640 1 -- 172.21.15.114:0/1448943860 shutdown_connections 2024-09-20T19:39:28.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:28.944+0000 7ff7c6c4f640 1 -- 172.21.15.114:0/1448943860 wait complete. 2024-09-20T19:39:29.613 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:29 smithi114 ceph-mon[28260]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:29.613 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:29 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:29.613 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:29 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:29.639 DEBUG:teuthology.orchestra.run.smithi114:osd.0> sudo journalctl -f -n 0 -u ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@osd.0.service 2024-09-20T19:39:29.641 INFO:tasks.cephadm:Deploying osd.1 on smithi114 with /dev/vg_nvme/lv_3... 2024-09-20T19:39:29.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T19:39:29.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:29 smithi160 ceph-mon[35859]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:29.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:29 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:29.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:29 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:30.271 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:39:30.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:39:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:39:30.910+0000 7f137ad23640 -1 osd.0 0 log_to_monitors true 2024-09-20T19:39:31.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:31 smithi114 ceph-mon[28260]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:31.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:31 smithi114 ceph-mon[28260]: from='osd.0 [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T19:39:31.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:31 smithi160 ceph-mon[35859]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:31.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:31 smithi160 ceph-mon[35859]: from='osd.0 [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T19:39:32.627 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:32 smithi114 ceph-mon[28260]: from='osd.0 [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:39:32.627 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:32 smithi114 ceph-mon[28260]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T19:39:32.628 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:32 smithi114 ceph-mon[28260]: from='osd.0 [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-20T19:39:32.628 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:32 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:39:32.628 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:32 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:32.628 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:32 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:32.628 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:32 smithi114 ceph-mon[28260]: from='osd.0 [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-20T19:39:32.628 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:32 smithi114 ceph-mon[28260]: osdmap e8: 1 total, 0 up, 1 in 2024-09-20T19:39:32.628 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:32 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:39:32.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:39:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:39:32.302+0000 7f1376483640 -1 osd.0 0 waiting for initial osdmap 2024-09-20T19:39:32.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:39:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:39:32.309+0000 7f1372abd640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:39:32.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:32 smithi160 ceph-mon[35859]: from='osd.0 [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T19:39:32.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:32 smithi160 ceph-mon[35859]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T19:39:32.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:32 smithi160 ceph-mon[35859]: from='osd.0 [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-20T19:39:32.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:32 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:39:32.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:32 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:32.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:32 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:32.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:32 smithi160 ceph-mon[35859]: from='osd.0 [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-20T19:39:32.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:32 smithi160 ceph-mon[35859]: osdmap e8: 1 total, 0 up, 1 in 2024-09-20T19:39:32.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:32 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:39:33.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:33 smithi114 ceph-mon[28260]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:33.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:33 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:39:33.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:33 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:39:33.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:33 smithi160 ceph-mon[35859]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:33.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:33 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:39:33.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:33 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:39:33.790 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:39:33.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi114:vg_nvme/lv_3 2024-09-20T19:39:34.313 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:39:34.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:34 smithi114 ceph-mon[28260]: purged_snaps scrub starts 2024-09-20T19:39:34.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:34 smithi114 ceph-mon[28260]: purged_snaps scrub ok 2024-09-20T19:39:34.521 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:34 smithi114 ceph-mon[28260]: osd.0 [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375] boot 2024-09-20T19:39:34.521 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:34 smithi114 ceph-mon[28260]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T19:39:34.521 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:34 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:39:34.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:34 smithi160 ceph-mon[35859]: purged_snaps scrub starts 2024-09-20T19:39:34.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:34 smithi160 ceph-mon[35859]: purged_snaps scrub ok 2024-09-20T19:39:34.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:34 smithi160 ceph-mon[35859]: osd.0 [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375] boot 2024-09-20T19:39:34.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:34 smithi160 ceph-mon[35859]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T19:39:34.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:34 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T19:39:35.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.166+0000 7fbbdf857640 1 -- 172.21.15.114:0/2067649899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd8076220 msgr2=0x7fbbd8074680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:39:35.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.166+0000 7fbbdf857640 1 --2- 172.21.15.114:0/2067649899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd8076220 0x7fbbd8074680 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4009880 tx=0x7fbbd402a4a0 comp rx=0 tx=0).stop 2024-09-20T19:39:35.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.166+0000 7fbbdf857640 1 -- 172.21.15.114:0/2067649899 shutdown_connections 2024-09-20T19:39:35.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.166+0000 7fbbdf857640 1 --2- 172.21.15.114:0/2067649899 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbbd8074c50 0x7fbbd80750b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:39:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.166+0000 7fbbdf857640 1 --2- 172.21.15.114:0/2067649899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd8076220 0x7fbbd8074680 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:39:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.166+0000 7fbbdf857640 1 -- 172.21.15.114:0/2067649899 >> 172.21.15.114:0/2067649899 conn(0x7fbbd80fd0c0 msgr2=0x7fbbd80ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:39:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.166+0000 7fbbdf857640 1 -- 172.21.15.114:0/2067649899 shutdown_connections 2024-09-20T19:39:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.166+0000 7fbbdf857640 1 -- 172.21.15.114:0/2067649899 wait complete. 2024-09-20T19:39:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.167+0000 7fbbdf857640 1 Processor -- start 2024-09-20T19:39:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.167+0000 7fbbdf857640 1 -- start start 2024-09-20T19:39:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.167+0000 7fbbdf857640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbbd8074c50 0x7fbbd819dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:39:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.167+0000 7fbbdf857640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd8076220 0x7fbbd819e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:39:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.167+0000 7fbbdf857640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd8110d90 con 0x7fbbd8076220 2024-09-20T19:39:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.167+0000 7fbbdf857640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd8110c10 con 0x7fbbd8074c50 2024-09-20T19:39:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.167+0000 7fbbdcdf2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd8076220 0x7fbbd819e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:39:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.167+0000 7fbbdcdf2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd8076220 0x7fbbd819e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55390/0 (socket says 172.21.15.114:55390) 2024-09-20T19:39:35.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.167+0000 7fbbdcdf2640 1 -- 172.21.15.114:0/3980590608 learned_addr learned my addr 172.21.15.114:0/3980590608 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:39:35.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.168+0000 7fbbdd5f3640 1 --2- 172.21.15.114:0/3980590608 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbbd8074c50 0x7fbbd819dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:39:35.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.168+0000 7fbbdcdf2640 1 -- 172.21.15.114:0/3980590608 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbbd8074c50 msgr2=0x7fbbd819dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:39:35.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.168+0000 7fbbdcdf2640 1 --2- 172.21.15.114:0/3980590608 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbbd8074c50 0x7fbbd819dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:39:35.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.168+0000 7fbbdcdf2640 1 -- 172.21.15.114:0/3980590608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd819e690 con 0x7fbbd8076220 2024-09-20T19:39:35.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.168+0000 7fbbdcdf2640 1 --2- 172.21.15.114:0/3980590608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd8076220 0x7fbbd819e010 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbbc800d6f0 tx=0x7fbbc800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:39:35.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.168+0000 7fbbce7fc640 1 -- 172.21.15.114:0/3980590608 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fbbc8017070 con 0x7fbbd8076220 2024-09-20T19:39:35.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.168+0000 7fbbdf857640 1 -- 172.21.15.114:0/3980590608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbd81a30d0 con 0x7fbbd8076220 2024-09-20T19:39:35.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.168+0000 7fbbce7fc640 1 -- 172.21.15.114:0/3980590608 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbbc800de00 con 0x7fbbd8076220 2024-09-20T19:39:35.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.168+0000 7fbbce7fc640 1 -- 172.21.15.114:0/3980590608 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fbbc8004bb0 con 0x7fbbd8076220 2024-09-20T19:39:35.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.168+0000 7fbbdf857640 1 -- 172.21.15.114:0/3980590608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbd81a3660 con 0x7fbbd8076220 2024-09-20T19:39:35.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.169+0000 7fbbdf857640 1 -- 172.21.15.114:0/3980590608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbd80750b0 con 0x7fbbd8076220 2024-09-20T19:39:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.170+0000 7fbbce7fc640 1 -- 172.21.15.114:0/3980590608 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 92497+0+0 (secure 0 0 0) 0x7fbbc8004e60 con 0x7fbbd8076220 2024-09-20T19:39:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.170+0000 7fbbce7fc640 1 --2- 172.21.15.114:0/3980590608 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbbc406e230 0x7fbbc40706f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:39:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.171+0000 7fbbdd5f3640 1 --2- 172.21.15.114:0/3980590608 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbbc406e230 0x7fbbc40706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:39:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.171+0000 7fbbce7fc640 1 -- 172.21.15.114:0/3980590608 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1982+0+0 (secure 0 0 0) 0x7fbbc808cea0 con 0x7fbbd8076220 2024-09-20T19:39:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.171+0000 7fbbdd5f3640 1 --2- 172.21.15.114:0/3980590608 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbbc406e230 0x7fbbc40706f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4009230 tx=0x7fbbd40058d0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:39:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.175+0000 7fbbce7fc640 1 -- 172.21.15.114:0/3980590608 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fbbc805f290 con 0x7fbbd8076220 2024-09-20T19:39:35.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:35.360+0000 7fbbdf857640 1 -- 172.21.15.114:0/3980590608 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fbbd8062630 con 0x7fbbc406e230 2024-09-20T19:39:35.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:35 smithi114 ceph-mon[28260]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:35.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:35 smithi114 ceph-mon[28260]: osdmap e10: 1 total, 1 up, 1 in 2024-09-20T19:39:35.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:35 smithi160 ceph-mon[35859]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T19:39:35.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:35 smithi160 ceph-mon[35859]: osdmap e10: 1 total, 1 up, 1 in 2024-09-20T19:39:36.451 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:36 smithi114 ceph-mon[28260]: from='client.14348 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:39:36.451 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:39:36.451 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:39:36.451 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:36.451 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:36.451 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:36.452 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:39:36.452 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:36.452 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:36.452 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:39:36.452 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:36.452 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:39:36.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:36 smithi160 ceph-mon[35859]: from='client.14348 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:39:36.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:39:36.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:39:36.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:36.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:36.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:36.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:39:36.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:36.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:36.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:39:36.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:36.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:39:37.544 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:37 smithi160 ceph-mon[35859]: Detected new or changed devices on smithi114 2024-09-20T19:39:37.544 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:37 smithi160 ceph-mon[35859]: Adjusting osd_memory_target on smithi114 to 11840M 2024-09-20T19:39:37.544 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:37 smithi160 ceph-mon[35859]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:37.592 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:37 smithi114 ceph-mon[28260]: Detected new or changed devices on smithi114 2024-09-20T19:39:37.592 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:37 smithi114 ceph-mon[28260]: Adjusting osd_memory_target on smithi114 to 11840M 2024-09-20T19:39:37.592 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:37 smithi114 ceph-mon[28260]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:39.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:39 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:39.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:39 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:39.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:39 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/2290482505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98aef96e-fc81-45ad-9cc1-49ce9ba617b5"}]: dispatch 2024-09-20T19:39:39.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:39 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/2290482505' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98aef96e-fc81-45ad-9cc1-49ce9ba617b5"}]': finished 2024-09-20T19:39:39.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:39 smithi114 ceph-mon[28260]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T19:39:39.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:39 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:39:39.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:39 smithi114 ceph-mon[28260]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:39.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:39 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/244870991' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:39:39.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:39 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:39.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:39 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:39.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:39 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/2290482505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98aef96e-fc81-45ad-9cc1-49ce9ba617b5"}]: dispatch 2024-09-20T19:39:39.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:39 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/2290482505' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98aef96e-fc81-45ad-9cc1-49ce9ba617b5"}]': finished 2024-09-20T19:39:39.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:39 smithi160 ceph-mon[35859]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T19:39:39.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:39 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:39:39.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:39 smithi160 ceph-mon[35859]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:39.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:39 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/244870991' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:39:41.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:41 smithi114 ceph-mon[28260]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:41.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:41 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:39:41.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:41 smithi160 ceph-mon[35859]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:41.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:41 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:39:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:42 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:42 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:42 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:42 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:39:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:42 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:42 smithi114 ceph-mon[28260]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:43.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:42 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:43.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:42 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:43.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:42 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:43.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:42 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:39:43.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:42 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:43.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:42 smithi160 ceph-mon[35859]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:45.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:45 smithi114 ceph-mon[28260]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:45.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:45 smithi160 ceph-mon[35859]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:47.485 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:47 smithi114 ceph-mon[28260]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:47.485 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T19:39:47.485 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:47.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:47 smithi160 ceph-mon[35859]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:47.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T19:39:47.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:39:48.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:48 smithi114 ceph-mon[28260]: Deploying daemon osd.1 on smithi114 2024-09-20T19:39:48.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:48 smithi160 ceph-mon[35859]: Deploying daemon osd.1 on smithi114 2024-09-20T19:39:49.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:49 smithi114 ceph-mon[28260]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:49.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:49 smithi160 ceph-mon[35859]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:51.623 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:51 smithi114 ceph-mon[28260]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:51.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:51 smithi160 ceph-mon[35859]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:53.579 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:53 smithi114 ceph-mon[28260]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:53.579 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:53 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:39:53.579 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:53 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:53.579 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:53 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:53.716 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:53 smithi160 ceph-mon[35859]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:53.717 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:53 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:39:53.717 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:53 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:53.717 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:53 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:55.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:55 smithi114 ceph-mon[28260]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:55.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:55 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:55.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:55 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:55.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:55 smithi160 ceph-mon[35859]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:55.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:55 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:55.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:55 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:56.585 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:56 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:39:56.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:56 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:39:57.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:57.542+0000 7fbbce7fc640 1 -- 172.21.15.114:0/3980590608 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbbd8062630 con 0x7fbbc406e230 2024-09-20T19:39:57.543 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 1 on host 'smithi114' 2024-09-20T19:39:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:57.544+0000 7fbbdf857640 1 -- 172.21.15.114:0/3980590608 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbbc406e230 msgr2=0x7fbbc40706f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:39:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:57.544+0000 7fbbdf857640 1 --2- 172.21.15.114:0/3980590608 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbbc406e230 0x7fbbc40706f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4009230 tx=0x7fbbd40058d0 comp rx=0 tx=0).stop 2024-09-20T19:39:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:57.544+0000 7fbbdf857640 1 -- 172.21.15.114:0/3980590608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd8076220 msgr2=0x7fbbd819e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:39:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:57.544+0000 7fbbdf857640 1 --2- 172.21.15.114:0/3980590608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd8076220 0x7fbbd819e010 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbbc800d6f0 tx=0x7fbbc800dbb0 comp rx=0 tx=0).stop 2024-09-20T19:39:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:57.544+0000 7fbbdf857640 1 -- 172.21.15.114:0/3980590608 shutdown_connections 2024-09-20T19:39:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:57.544+0000 7fbbdf857640 1 --2- 172.21.15.114:0/3980590608 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbbc406e230 0x7fbbc40706f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:39:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:57.544+0000 7fbbdf857640 1 --2- 172.21.15.114:0/3980590608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd8076220 0x7fbbd819e010 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:39:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:57.544+0000 7fbbdf857640 1 --2- 172.21.15.114:0/3980590608 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbbd8074c50 0x7fbbd819dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:39:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:57.544+0000 7fbbdf857640 1 -- 172.21.15.114:0/3980590608 >> 172.21.15.114:0/3980590608 conn(0x7fbbd80fd0c0 msgr2=0x7fbbd8103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:39:57.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:57.544+0000 7fbbdf857640 1 -- 172.21.15.114:0/3980590608 shutdown_connections 2024-09-20T19:39:57.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:39:57.545+0000 7fbbdf857640 1 -- 172.21.15.114:0/3980590608 wait complete. 2024-09-20T19:39:57.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:57 smithi114 ceph-mon[28260]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:57.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:57 smithi160 ceph-mon[35859]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:58.145 DEBUG:teuthology.orchestra.run.smithi114:osd.1> sudo journalctl -f -n 0 -u ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@osd.1.service 2024-09-20T19:39:58.147 INFO:tasks.cephadm:Deploying osd.2 on smithi114 with /dev/vg_nvme/lv_2... 2024-09-20T19:39:58.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T19:39:58.851 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:39:58.932 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:58 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:58.932 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:58 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:58.932 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:58 smithi114 ceph-mon[28260]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:59.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:58 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:59.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:58 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:39:59.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:58 smithi160 ceph-mon[35859]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:39:59.198 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:39:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:39:58.930+0000 7fcc6a202640 -1 osd.1 0 log_to_monitors true 2024-09-20T19:39:59.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:39:59 smithi160 ceph-mon[35859]: from='osd.1 [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T19:39:59.809 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:39:59 smithi114 ceph-mon[28260]: from='osd.1 [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T19:40:00.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:00 smithi160 ceph-mon[35859]: from='osd.1 [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:40:00.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:00 smithi160 ceph-mon[35859]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T19:40:00.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:00 smithi160 ceph-mon[35859]: from='osd.1 [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-20T19:40:00.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:00 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:40:00.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:00 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:00.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:00 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:00.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:00 smithi160 ceph-mon[35859]: overall HEALTH_OK 2024-09-20T19:40:00.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:00 smithi160 ceph-mon[35859]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:40:00.850 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:00 smithi114 ceph-mon[28260]: from='osd.1 [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T19:40:00.850 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:00 smithi114 ceph-mon[28260]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T19:40:00.851 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:00 smithi114 ceph-mon[28260]: from='osd.1 [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-20T19:40:00.851 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:00 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:40:00.851 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:00 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:00.851 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:00 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:00.851 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:00 smithi114 ceph-mon[28260]: overall HEALTH_OK 2024-09-20T19:40:00.851 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:00 smithi114 ceph-mon[28260]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:40:01.635 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:01 smithi114 ceph-mon[28260]: from='osd.1 [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-20T19:40:01.635 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:01 smithi114 ceph-mon[28260]: osdmap e13: 2 total, 1 up, 2 in 2024-09-20T19:40:01.636 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:01 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:40:01.636 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:01 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:40:01.636 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:01 smithi114 ceph-mon[28260]: from='osd.1 [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805]' entity='osd.1' 2024-09-20T19:40:01.636 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:01 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:40:01.636 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:40:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:40:01.334+0000 7fcc66999640 -1 osd.1 0 waiting for initial osdmap 2024-09-20T19:40:01.636 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:40:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:40:01.341+0000 7fcc5df94640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:40:02.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:01 smithi160 ceph-mon[35859]: from='osd.1 [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-20T19:40:02.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:01 smithi160 ceph-mon[35859]: osdmap e13: 2 total, 1 up, 2 in 2024-09-20T19:40:02.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:01 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:40:02.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:01 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:40:02.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:01 smithi160 ceph-mon[35859]: from='osd.1 [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805]' entity='osd.1' 2024-09-20T19:40:02.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:01 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:40:02.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:02 smithi114 ceph-mon[28260]: purged_snaps scrub starts 2024-09-20T19:40:02.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:02 smithi114 ceph-mon[28260]: purged_snaps scrub ok 2024-09-20T19:40:02.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:02 smithi114 ceph-mon[28260]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:40:02.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:02 smithi114 ceph-mon[28260]: osd.1 [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805] boot 2024-09-20T19:40:02.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:02 smithi114 ceph-mon[28260]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T19:40:02.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:02 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:40:02.878 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:40:02.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi114:vg_nvme/lv_2 2024-09-20T19:40:03.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:02 smithi160 ceph-mon[35859]: purged_snaps scrub starts 2024-09-20T19:40:03.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:02 smithi160 ceph-mon[35859]: purged_snaps scrub ok 2024-09-20T19:40:03.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:02 smithi160 ceph-mon[35859]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T19:40:03.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:02 smithi160 ceph-mon[35859]: osd.1 [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805] boot 2024-09-20T19:40:03.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:02 smithi160 ceph-mon[35859]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T19:40:03.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:02 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T19:40:03.422 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:40:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.949+0000 7f3c7b08b640 1 -- 172.21.15.114:0/280920826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c741023d0 msgr2=0x7f3c741047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:40:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.949+0000 7f3c7b08b640 1 --2- 172.21.15.114:0/280920826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c741023d0 0x7f3c741047c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3c680098b0 tx=0x7f3c6802a4d0 comp rx=0 tx=0).stop 2024-09-20T19:40:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.949+0000 7f3c7b08b640 1 -- 172.21.15.114:0/280920826 shutdown_connections 2024-09-20T19:40:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.949+0000 7f3c7b08b640 1 --2- 172.21.15.114:0/280920826 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3c74104d00 0x7f3c741070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:40:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.949+0000 7f3c7b08b640 1 --2- 172.21.15.114:0/280920826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c741023d0 0x7f3c741047c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:40:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.949+0000 7f3c7b08b640 1 -- 172.21.15.114:0/280920826 >> 172.21.15.114:0/280920826 conn(0x7f3c740fbff0 msgr2=0x7f3c740fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:40:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.949+0000 7f3c7b08b640 1 -- 172.21.15.114:0/280920826 shutdown_connections 2024-09-20T19:40:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.950+0000 7f3c7b08b640 1 -- 172.21.15.114:0/280920826 wait complete. 2024-09-20T19:40:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.950+0000 7f3c7b08b640 1 Processor -- start 2024-09-20T19:40:03.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.950+0000 7f3c7b08b640 1 -- start start 2024-09-20T19:40:03.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.950+0000 7f3c7b08b640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3c741023d0 0x7f3c74199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:40:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.950+0000 7f3c7b08b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c74104d00 0x7f3c74199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:40:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.950+0000 7f3c7b08b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7410a810 con 0x7f3c74104d00 2024-09-20T19:40:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.951+0000 7f3c7b08b640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7410a690 con 0x7f3c741023d0 2024-09-20T19:40:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.951+0000 7f3c73fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c74104d00 0x7f3c74199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:40:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.951+0000 7f3c73fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c74104d00 0x7f3c74199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54040/0 (socket says 172.21.15.114:54040) 2024-09-20T19:40:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.951+0000 7f3c73fff640 1 -- 172.21.15.114:0/3189914189 learned_addr learned my addr 172.21.15.114:0/3189914189 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:40:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.951+0000 7f3c78e27640 1 --2- 172.21.15.114:0/3189914189 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3c741023d0 0x7f3c74199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:40:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.951+0000 7f3c73fff640 1 -- 172.21.15.114:0/3189914189 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3c741023d0 msgr2=0x7f3c74199770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:40:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.951+0000 7f3c73fff640 1 --2- 172.21.15.114:0/3189914189 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3c741023d0 0x7f3c74199770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:40:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.951+0000 7f3c73fff640 1 -- 172.21.15.114:0/3189914189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c7419a330 con 0x7f3c74104d00 2024-09-20T19:40:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.951+0000 7f3c73fff640 1 --2- 172.21.15.114:0/3189914189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c74104d00 0x7f3c74199cb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000b3d0 tx=0x7f3c6000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:40:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.951+0000 7f3c71ffb640 1 -- 172.21.15.114:0/3189914189 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3c60015020 con 0x7f3c74104d00 2024-09-20T19:40:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.951+0000 7f3c71ffb640 1 -- 172.21.15.114:0/3189914189 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3c60011af0 con 0x7f3c74104d00 2024-09-20T19:40:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.951+0000 7f3c7b08b640 1 -- 172.21.15.114:0/3189914189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c7419ed70 con 0x7f3c74104d00 2024-09-20T19:40:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.951+0000 7f3c71ffb640 1 -- 172.21.15.114:0/3189914189 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3c60010d30 con 0x7f3c74104d00 2024-09-20T19:40:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.952+0000 7f3c7b08b640 1 -- 172.21.15.114:0/3189914189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c7419f2f0 con 0x7f3c74104d00 2024-09-20T19:40:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.953+0000 7f3c7b08b640 1 -- 172.21.15.114:0/3189914189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c741027b0 con 0x7f3c74104d00 2024-09-20T19:40:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.953+0000 7f3c71ffb640 1 -- 172.21.15.114:0/3189914189 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 92497+0+0 (secure 0 0 0) 0x7f3c60018020 con 0x7f3c74104d00 2024-09-20T19:40:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.954+0000 7f3c71ffb640 1 --2- 172.21.15.114:0/3189914189 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f3c6406e280 0x7f3c64070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:40:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.954+0000 7f3c71ffb640 1 -- 172.21.15.114:0/3189914189 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2414+0+0 (secure 0 0 0) 0x7f3c6008ce70 con 0x7f3c74104d00 2024-09-20T19:40:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.954+0000 7f3c78e27640 1 --2- 172.21.15.114:0/3189914189 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f3c6406e280 0x7f3c64070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:40:03.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.954+0000 7f3c78e27640 1 --2- 172.21.15.114:0/3189914189 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f3c6406e280 0x7f3c64070740 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3c68000c00 tx=0x7f3c68005a20 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:40:03.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:03.958+0000 7f3c71ffb640 1 -- 172.21.15.114:0/3189914189 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f3c6001f020 con 0x7f3c74104d00 2024-09-20T19:40:04.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:04.138+0000 7f3c7b08b640 1 -- 172.21.15.114:0/3189914189 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3c74062630 con 0x7f3c6406e280 2024-09-20T19:40:04.670 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:04 smithi114 ceph-mon[28260]: osdmap e15: 2 total, 2 up, 2 in 2024-09-20T19:40:04.670 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:04 smithi114 ceph-mon[28260]: from='client.14366 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:40:04.670 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:04 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:40:04.670 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:04 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:40:04.670 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:04 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:04.670 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:04 smithi114 ceph-mon[28260]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:05.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:04 smithi160 ceph-mon[35859]: osdmap e15: 2 total, 2 up, 2 in 2024-09-20T19:40:05.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:04 smithi160 ceph-mon[35859]: from='client.14366 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:40:05.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:04 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:40:05.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:04 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:40:05.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:04 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:05.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:04 smithi160 ceph-mon[35859]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:05.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:05 smithi114 ceph-mon[28260]: Detected new or changed devices on smithi114 2024-09-20T19:40:05.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:05 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:05.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:05 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:05.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:05 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:40:05.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:05 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:40:05.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:05 smithi114 ceph-mon[28260]: Adjusting osd_memory_target on smithi114 to 5920M 2024-09-20T19:40:05.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:05 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:05.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:05 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:05.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:05 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:40:05.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:05 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:05.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:05 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:40:06.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:05 smithi160 ceph-mon[35859]: Detected new or changed devices on smithi114 2024-09-20T19:40:06.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:05 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:06.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:05 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:06.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:05 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:40:06.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:05 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:40:06.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:05 smithi160 ceph-mon[35859]: Adjusting osd_memory_target on smithi114 to 5920M 2024-09-20T19:40:06.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:05 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:06.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:05 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:06.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:05 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:40:06.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:05 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:06.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:05 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:40:07.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:06 smithi160 ceph-mon[35859]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:07.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:06 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:07.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:06 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:07.166 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:06 smithi114 ceph-mon[28260]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:07.167 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:06 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:07.167 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:06 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:07.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:07 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1280211418' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89c2fdec-88bf-4bcb-94be-fef898abddbd"}]: dispatch 2024-09-20T19:40:07.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:07 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1280211418' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89c2fdec-88bf-4bcb-94be-fef898abddbd"}]': finished 2024-09-20T19:40:07.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:07 smithi114 ceph-mon[28260]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T19:40:07.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:07 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:40:07.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:07 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3104750589' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:40:08.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:07 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1280211418' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89c2fdec-88bf-4bcb-94be-fef898abddbd"}]: dispatch 2024-09-20T19:40:08.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:07 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1280211418' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89c2fdec-88bf-4bcb-94be-fef898abddbd"}]': finished 2024-09-20T19:40:08.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:07 smithi160 ceph-mon[35859]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T19:40:08.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:07 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:40:08.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:07 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/3104750589' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:40:08.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:08 smithi160 ceph-mon[35859]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:09.050 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:08 smithi114 ceph-mon[28260]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:11 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:11 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:11 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:11 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:40:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:11 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:11 smithi114 ceph-mon[28260]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:11.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:11 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:11.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:11 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:11.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:11 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:11.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:11 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:40:11.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:11 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:11.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:11 smithi160 ceph-mon[35859]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:12.360 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:12 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:40:12.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:12 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:40:13.328 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:13 smithi114 ceph-mon[28260]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:13.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:13 smithi160 ceph-mon[35859]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:15.613 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:15 smithi114 ceph-mon[28260]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:15.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:15 smithi160 ceph-mon[35859]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:16.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:16 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T19:40:16.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:16 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:16.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:16 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T19:40:16.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:16 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:17.478 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:17 smithi114 ceph-mon[28260]: Deploying daemon osd.2 on smithi114 2024-09-20T19:40:17.478 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:17 smithi114 ceph-mon[28260]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:17.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:17 smithi160 ceph-mon[35859]: Deploying daemon osd.2 on smithi114 2024-09-20T19:40:17.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:17 smithi160 ceph-mon[35859]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:19.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:19 smithi114 ceph-mon[28260]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:19.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:19 smithi160 ceph-mon[35859]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:21.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:21 smithi114 ceph-mon[28260]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:21.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:21 smithi160 ceph-mon[35859]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:22.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:22 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:40:22.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:22 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:22.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:22 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:22.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:22 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:40:22.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:22 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:22.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:22 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:23.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:23 smithi114 ceph-mon[28260]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:23.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:23 smithi160 ceph-mon[35859]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:24.546 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 2 on host 'smithi114' 2024-09-20T19:40:24.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:24.544+0000 7f3c71ffb640 1 -- 172.21.15.114:0/3189914189 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3c74062630 con 0x7f3c6406e280 2024-09-20T19:40:24.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:24.547+0000 7f3c7b08b640 1 -- 172.21.15.114:0/3189914189 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f3c6406e280 msgr2=0x7f3c64070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:40:24.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:24.547+0000 7f3c7b08b640 1 --2- 172.21.15.114:0/3189914189 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f3c6406e280 0x7f3c64070740 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3c68000c00 tx=0x7f3c68005a20 comp rx=0 tx=0).stop 2024-09-20T19:40:24.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:24.547+0000 7f3c7b08b640 1 -- 172.21.15.114:0/3189914189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c74104d00 msgr2=0x7f3c74199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:40:24.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:24.547+0000 7f3c7b08b640 1 --2- 172.21.15.114:0/3189914189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c74104d00 0x7f3c74199cb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000b3d0 tx=0x7f3c6000b890 comp rx=0 tx=0).stop 2024-09-20T19:40:24.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:24.547+0000 7f3c7b08b640 1 -- 172.21.15.114:0/3189914189 shutdown_connections 2024-09-20T19:40:24.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:24.547+0000 7f3c7b08b640 1 --2- 172.21.15.114:0/3189914189 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f3c6406e280 0x7f3c64070740 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:40:24.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:24.547+0000 7f3c7b08b640 1 --2- 172.21.15.114:0/3189914189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c74104d00 0x7f3c74199cb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:40:24.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:24.547+0000 7f3c7b08b640 1 --2- 172.21.15.114:0/3189914189 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3c741023d0 0x7f3c74199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:40:24.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:24.547+0000 7f3c7b08b640 1 -- 172.21.15.114:0/3189914189 >> 172.21.15.114:0/3189914189 conn(0x7f3c740fbff0 msgr2=0x7f3c74101aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:40:24.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:24.547+0000 7f3c7b08b640 1 -- 172.21.15.114:0/3189914189 shutdown_connections 2024-09-20T19:40:24.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:40:24.547+0000 7f3c7b08b640 1 -- 172.21.15.114:0/3189914189 wait complete. 2024-09-20T19:40:25.229 DEBUG:teuthology.orchestra.run.smithi114:osd.2> sudo journalctl -f -n 0 -u ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@osd.2.service 2024-09-20T19:40:25.231 INFO:tasks.cephadm:Deploying osd.3 on smithi160 with /dev/vg_nvme/lv_4... 2024-09-20T19:40:25.232 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T19:40:25.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:25 smithi114 ceph-mon[28260]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:25.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:25.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:25 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:25.493 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi160/config 2024-09-20T19:40:25.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:25 smithi160 ceph-mon[35859]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:25.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:25.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:25 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:26.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:26 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:40:26.771 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:26 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:40:27.667 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:27 smithi114 ceph-mon[28260]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:27.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:40:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:40:27.443+0000 7f767a3fd640 -1 osd.2 0 log_to_monitors true 2024-09-20T19:40:27.702 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:27 smithi160 ceph-mon[35859]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:27.980 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:40:28.011 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi160:vg_nvme/lv_4 2024-09-20T19:40:28.262 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi160/config 2024-09-20T19:40:28.557 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:28 smithi160 ceph-mon[35859]: from='osd.2 [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T19:40:28.557 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:28 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:28.557 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:28 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:28.669 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:28 smithi114 ceph-mon[28260]: from='osd.2 [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T19:40:28.670 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:28 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:28.670 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:28 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:28.723 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.720+0000 7f86c071c640 1 -- 172.21.15.160:0/3989270413 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f86b8100880 msgr2=0x7f86b8100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:40:28.723 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.720+0000 7f86c071c640 1 --2- 172.21.15.160:0/3989270413 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f86b8100880 0x7f86b8100c80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f86b4009880 tx=0x7f86b402a4a0 comp rx=0 tx=0).stop 2024-09-20T19:40:28.723 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.720+0000 7f86c071c640 1 -- 172.21.15.160:0/3989270413 shutdown_connections 2024-09-20T19:40:28.723 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.720+0000 7f86c071c640 1 --2- 172.21.15.160:0/3989270413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86b8101a80 0x7f86b8101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:40:28.723 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.720+0000 7f86c071c640 1 --2- 172.21.15.160:0/3989270413 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f86b8100880 0x7f86b8100c80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:40:28.724 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.720+0000 7f86c071c640 1 -- 172.21.15.160:0/3989270413 >> 172.21.15.160:0/3989270413 conn(0x7f86b80fc010 msgr2=0x7f86b80fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:40:28.724 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.720+0000 7f86c071c640 1 -- 172.21.15.160:0/3989270413 shutdown_connections 2024-09-20T19:40:28.724 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.720+0000 7f86c071c640 1 -- 172.21.15.160:0/3989270413 wait complete. 2024-09-20T19:40:28.724 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.721+0000 7f86c071c640 1 Processor -- start 2024-09-20T19:40:28.724 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.721+0000 7f86c071c640 1 -- start start 2024-09-20T19:40:28.724 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.721+0000 7f86c071c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86b8100880 0x7f86b81974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:40:28.724 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.721+0000 7f86c071c640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f86b8101a80 0x7f86b8197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:40:28.724 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.721+0000 7f86c071c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f86b81096a0 con 0x7f86b8100880 2024-09-20T19:40:28.724 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.721+0000 7f86c071c640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f86b8109520 con 0x7f86b8101a80 2024-09-20T19:40:28.724 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.721+0000 7f86bdcb7640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f86b8101a80 0x7f86b8197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:40:28.724 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.721+0000 7f86bdcb7640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f86b8101a80 0x7f86b8197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:56426/0 (socket says 172.21.15.160:56426) 2024-09-20T19:40:28.724 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.721+0000 7f86bdcb7640 1 -- 172.21.15.160:0/3111147641 learned_addr learned my addr 172.21.15.160:0/3111147641 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-09-20T19:40:28.725 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.721+0000 7f86be4b8640 1 --2- 172.21.15.160:0/3111147641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86b8100880 0x7f86b81974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:40:28.725 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.721+0000 7f86bdcb7640 1 -- 172.21.15.160:0/3111147641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86b8100880 msgr2=0x7f86b81974e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:40:28.725 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.721+0000 7f86bdcb7640 1 --2- 172.21.15.160:0/3111147641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86b8100880 0x7f86b81974e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:40:28.725 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.721+0000 7f86bdcb7640 1 -- 172.21.15.160:0/3111147641 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b8198010 con 0x7f86b8101a80 2024-09-20T19:40:28.725 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.721+0000 7f86bdcb7640 1 --2- 172.21.15.160:0/3111147641 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f86b8101a80 0x7f86b8197a20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f86a800c8a0 tx=0x7f86a800cd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:40:28.725 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.722+0000 7f86af7fe640 1 -- 172.21.15.160:0/3111147641 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f86a80079b0 con 0x7f86b8101a80 2024-09-20T19:40:28.725 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.722+0000 7f86af7fe640 1 -- 172.21.15.160:0/3111147641 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f86a8007b50 con 0x7f86b8101a80 2024-09-20T19:40:28.725 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.722+0000 7f86c071c640 1 -- 172.21.15.160:0/3111147641 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86b819ca80 con 0x7f86b8101a80 2024-09-20T19:40:28.725 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.722+0000 7f86af7fe640 1 -- 172.21.15.160:0/3111147641 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f86a8018700 con 0x7f86b8101a80 2024-09-20T19:40:28.725 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.722+0000 7f86c071c640 1 -- 172.21.15.160:0/3111147641 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86b819cfe0 con 0x7f86b8101a80 2024-09-20T19:40:28.726 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.723+0000 7f86c071c640 1 -- 172.21.15.160:0/3111147641 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86b8100c80 con 0x7f86b8101a80 2024-09-20T19:40:28.726 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.723+0000 7f86af7fe640 1 -- 172.21.15.160:0/3111147641 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 18) ==== 92497+0+0 (secure 0 0 0) 0x7f86a8007d00 con 0x7f86b8101a80 2024-09-20T19:40:28.726 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.724+0000 7f86af7fe640 1 --2- 172.21.15.160:0/3111147641 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f86a406e280 0x7f86a4070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:40:28.726 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.724+0000 7f86af7fe640 1 -- 172.21.15.160:0/3111147641 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 2550+0+0 (secure 0 0 0) 0x7f86a808d720 con 0x7f86b8101a80 2024-09-20T19:40:28.726 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.724+0000 7f86be4b8640 1 --2- 172.21.15.160:0/3111147641 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f86a406e280 0x7f86a4070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:40:28.727 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.724+0000 7f86be4b8640 1 --2- 172.21.15.160:0/3111147641 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f86a406e280 0x7f86a4070740 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f86b4000c00 tx=0x7f86b4005a20 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:40:28.729 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.728+0000 7f86af7fe640 1 -- 172.21.15.160:0/3111147641 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f86a8014030 con 0x7f86b8101a80 2024-09-20T19:40:28.904 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:28.903+0000 7f86c071c640 1 -- 172.21.15.160:0/3111147641 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f86b806aa10 con 0x7f86a406e280 2024-09-20T19:40:29.633 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:29 smithi160 ceph-mon[35859]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:29.633 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:29 smithi160 ceph-mon[35859]: from='osd.2 [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T19:40:29.633 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:29 smithi160 ceph-mon[35859]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T19:40:29.633 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:29 smithi160 ceph-mon[35859]: from='osd.2 [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-20T19:40:29.633 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:29 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:40:29.634 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:29 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:40:29.634 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:29 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:40:29.634 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:29 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:29.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:40:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:40:29.431+0000 7f7675b5d640 -1 osd.2 0 waiting for initial osdmap 2024-09-20T19:40:29.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:40:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:40:29.440+0000 7f7672197640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:40:29.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:29 smithi114 ceph-mon[28260]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:29.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:29 smithi114 ceph-mon[28260]: from='osd.2 [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T19:40:29.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:29 smithi114 ceph-mon[28260]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T19:40:29.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:29 smithi114 ceph-mon[28260]: from='osd.2 [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-20T19:40:29.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:29 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:40:29.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:29 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:40:29.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:29 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:40:29.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:29 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:30.490 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:30 smithi160 ceph-mon[35859]: from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:40:30.490 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:30 smithi160 ceph-mon[35859]: from='osd.2 [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-20T19:40:30.490 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:30 smithi160 ceph-mon[35859]: osdmap e18: 3 total, 2 up, 3 in 2024-09-20T19:40:30.490 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:30 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:40:30.490 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:30 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:40:30.491 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:30 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:40:30.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:30 smithi114 ceph-mon[28260]: from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:40:30.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:30 smithi114 ceph-mon[28260]: from='osd.2 [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-20T19:40:30.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:30 smithi114 ceph-mon[28260]: osdmap e18: 3 total, 2 up, 3 in 2024-09-20T19:40:30.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:30 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:40:30.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:30 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:40:30.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:30 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:40:31.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: purged_snaps scrub starts 2024-09-20T19:40:31.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: purged_snaps scrub ok 2024-09-20T19:40:31.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:31.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: osd.2 [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493] boot 2024-09-20T19:40:31.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T19:40:31.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:40:31.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:31.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:31.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:40:31.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:40:31.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:40:31.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:31.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:31.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:40:31.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:31.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/1058784380' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64f8a718-5814-4210-8696-b2acd153c046"}]: dispatch 2024-09-20T19:40:31.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64f8a718-5814-4210-8696-b2acd153c046"}]: dispatch 2024-09-20T19:40:31.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64f8a718-5814-4210-8696-b2acd153c046"}]': finished 2024-09-20T19:40:31.718 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: osdmap e20: 4 total, 3 up, 4 in 2024-09-20T19:40:31.718 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:31.718 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:31 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:40:31.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: purged_snaps scrub starts 2024-09-20T19:40:31.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: purged_snaps scrub ok 2024-09-20T19:40:31.772 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-20T19:40:31.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: osd.2 [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493] boot 2024-09-20T19:40:31.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T19:40:31.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T19:40:31.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:31.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:31.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:40:31.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:40:31.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:40:31.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:31.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:31.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:40:31.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:31.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/1058784380' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64f8a718-5814-4210-8696-b2acd153c046"}]: dispatch 2024-09-20T19:40:31.773 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64f8a718-5814-4210-8696-b2acd153c046"}]: dispatch 2024-09-20T19:40:31.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64f8a718-5814-4210-8696-b2acd153c046"}]': finished 2024-09-20T19:40:31.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: osdmap e20: 4 total, 3 up, 4 in 2024-09-20T19:40:31.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:31.774 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:31 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:40:32.800 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:32 smithi114 ceph-mon[28260]: Detected new or changed devices on smithi114 2024-09-20T19:40:32.800 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:32 smithi114 ceph-mon[28260]: Adjusting osd_memory_target on smithi114 to 3946M 2024-09-20T19:40:32.800 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:32 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/3841002509' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:40:32.800 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:32 smithi114 ceph-mon[28260]: osdmap e21: 4 total, 3 up, 4 in 2024-09-20T19:40:32.800 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:32 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:33.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:32 smithi160 ceph-mon[35859]: Detected new or changed devices on smithi114 2024-09-20T19:40:33.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:32 smithi160 ceph-mon[35859]: Adjusting osd_memory_target on smithi114 to 3946M 2024-09-20T19:40:33.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:32 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/3841002509' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:40:33.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:32 smithi160 ceph-mon[35859]: osdmap e21: 4 total, 3 up, 4 in 2024-09-20T19:40:33.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:32 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:33.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:33 smithi114 ceph-mon[28260]: pgmap v87: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:33.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:33 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-20T19:40:33.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:33 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T19:40:33.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:33 smithi114 ceph-mon[28260]: osdmap e22: 4 total, 3 up, 4 in 2024-09-20T19:40:33.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:33 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:33.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:33 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T19:40:34.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:33 smithi160 ceph-mon[35859]: pgmap v87: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:34.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:33 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-20T19:40:34.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:33 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T19:40:34.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:33 smithi160 ceph-mon[35859]: osdmap e22: 4 total, 3 up, 4 in 2024-09-20T19:40:34.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:33 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:34.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:33 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T19:40:34.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:40:34 smithi114 sudo[69502]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T19:40:34.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:40:34 smithi114 sudo[69502]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:40:34.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:40:34 smithi114 sudo[69502]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:40:34.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:40:34 smithi114 sudo[69502]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:40:35.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69506]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T19:40:35.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69506]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:40:35.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69506]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:40:35.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69506]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:40:35.327 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:35 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:40:35.327 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:35 smithi114 ceph-mon[28260]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:35.327 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:35 smithi114 ceph-mon[28260]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T19:40:35.327 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:35 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:35.327 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:35 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:35.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:35 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T19:40:35.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:35 smithi160 ceph-mon[35859]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:35.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:35 smithi160 ceph-mon[35859]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T19:40:35.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:35 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:35.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:35 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:35.591 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69622]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T19:40:35.591 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69622]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:40:35.591 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69622]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:40:35.591 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69622]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:40:35.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69664]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T19:40:35.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69664]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:40:35.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69664]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:40:35.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69664]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:40:36.104 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69699]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T19:40:36.104 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69699]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:40:36.104 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69699]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:40:36.104 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:40:35 smithi114 sudo[69699]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:40:36.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:40:36 smithi114 sudo[69702]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T19:40:36.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:40:36 smithi114 sudo[69702]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:40:36.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:40:36 smithi114 sudo[69702]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:40:36.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:40:36 smithi114 sudo[69702]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:40:36.364 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 sudo[69705]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T19:40:36.364 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 sudo[69705]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:40:36.364 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 sudo[69705]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:40:36.364 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 sudo[69705]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:40:36.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 sudo[69708]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T19:40:36.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 sudo[69708]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:40:36.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 sudo[69708]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:40:36.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 sudo[69708]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:40:36.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 ceph-mon[28260]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T19:40:36.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:36.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:36.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:36.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:36.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:40:36.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:36.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:36 smithi114 ceph-mon[28260]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:40:36.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:36 smithi160 ceph-mon[35859]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T19:40:36.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:36.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:36.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:36.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:36.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:40:36.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:36 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:36.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:36 smithi160 ceph-mon[35859]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:40:37.111 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:36 smithi160 sudo[41023]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T19:40:37.111 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:36 smithi160 sudo[41023]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:40:37.111 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:36 smithi160 sudo[41023]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:40:37.111 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:36 smithi160 sudo[41023]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:40:37.126 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:37.124+0000 7f86af7fe640 1 -- 172.21.15.160:0/3111147641 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f86a8057cb0 con 0x7f86b8101a80 2024-09-20T19:40:37.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:37 smithi160 sudo[41026]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T19:40:37.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:37 smithi160 sudo[41026]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T19:40:37.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:37 smithi160 sudo[41026]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T19:40:37.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:37 smithi160 sudo[41026]: pam_unix(sudo:session): session closed for user root 2024-09-20T19:40:37.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:37 smithi160 ceph-mon[35859]: pgmap v92: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:37.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:37 smithi160 ceph-mon[35859]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:40:37.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:37 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-09-20T19:40:37.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:37 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:40:37.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:37 smithi160 ceph-mon[35859]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:40:37.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:37 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-09-20T19:40:37.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:37 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:40:37.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:37 smithi160 ceph-mon[35859]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:40:37.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:37 smithi160 ceph-mon[35859]: mgrmap e19: smithi114.rvhgji(active, since 2m), standbys: smithi160.bspiak 2024-09-20T19:40:37.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:37 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:37.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:37 smithi114 ceph-mon[28260]: pgmap v92: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:37.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:37 smithi114 ceph-mon[28260]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:40:37.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:37 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-09-20T19:40:37.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:37 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:40:37.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:37 smithi114 ceph-mon[28260]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T19:40:37.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:37 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-09-20T19:40:37.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:37 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-09-20T19:40:37.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:37 smithi114 ceph-mon[28260]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T19:40:37.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:37 smithi114 ceph-mon[28260]: mgrmap e19: smithi114.rvhgji(active, since 2m), standbys: smithi160.bspiak 2024-09-20T19:40:37.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:37 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:38.755 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:38 smithi160 ceph-mon[35859]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:38.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:38 smithi114 ceph-mon[28260]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:40.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:39 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T19:40:40.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:39 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:40.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:39 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T19:40:40.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:39 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:41.028 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:40 smithi160 ceph-mon[35859]: Deploying daemon osd.3 on smithi160 2024-09-20T19:40:41.028 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:40 smithi160 ceph-mon[35859]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:40 smithi114 ceph-mon[28260]: Deploying daemon osd.3 on smithi160 2024-09-20T19:40:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:40 smithi114 ceph-mon[28260]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:42.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:41 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:40:42.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:41 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:40:43.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:42 smithi160 ceph-mon[35859]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:43.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:42 smithi114 ceph-mon[28260]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:45.467 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:45 smithi160 ceph-mon[35859]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:45.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:45 smithi114 ceph-mon[28260]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:46.587 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:46 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:40:46.588 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:46 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:46.588 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:46 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:46.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:46 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:40:46.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:46 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:46.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:46 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:47.411 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:47 smithi160 ceph-mon[35859]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:47.616 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:47 smithi114 ceph-mon[28260]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:49.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:49 smithi160 ceph-mon[35859]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:49.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:49 smithi114 ceph-mon[28260]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:50.155 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:50.154+0000 7f86af7fe640 1 -- 172.21.15.160:0/3111147641 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f86b806aa10 con 0x7f86a406e280 2024-09-20T19:40:50.156 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 3 on host 'smithi160' 2024-09-20T19:40:50.158 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:50.156+0000 7f86c071c640 1 -- 172.21.15.160:0/3111147641 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f86a406e280 msgr2=0x7f86a4070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:40:50.158 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:50.156+0000 7f86c071c640 1 --2- 172.21.15.160:0/3111147641 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f86a406e280 0x7f86a4070740 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f86b4000c00 tx=0x7f86b4005a20 comp rx=0 tx=0).stop 2024-09-20T19:40:50.158 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:50.156+0000 7f86c071c640 1 -- 172.21.15.160:0/3111147641 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f86b8101a80 msgr2=0x7f86b8197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:40:50.158 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:50.156+0000 7f86c071c640 1 --2- 172.21.15.160:0/3111147641 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f86b8101a80 0x7f86b8197a20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f86a800c8a0 tx=0x7f86a800cd60 comp rx=0 tx=0).stop 2024-09-20T19:40:50.159 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:50.156+0000 7f86c071c640 1 -- 172.21.15.160:0/3111147641 shutdown_connections 2024-09-20T19:40:50.159 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:50.156+0000 7f86c071c640 1 --2- 172.21.15.160:0/3111147641 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f86a406e280 0x7f86a4070740 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:40:50.159 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:50.156+0000 7f86c071c640 1 --2- 172.21.15.160:0/3111147641 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f86b8101a80 0x7f86b8197a20 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:40:50.159 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:50.156+0000 7f86c071c640 1 --2- 172.21.15.160:0/3111147641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86b8100880 0x7f86b81974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:40:50.159 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:50.156+0000 7f86c071c640 1 -- 172.21.15.160:0/3111147641 >> 172.21.15.160:0/3111147641 conn(0x7f86b80fc010 msgr2=0x7f86b806a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:40:50.159 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:50.157+0000 7f86c071c640 1 -- 172.21.15.160:0/3111147641 shutdown_connections 2024-09-20T19:40:50.159 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:50.157+0000 7f86c071c640 1 -- 172.21.15.160:0/3111147641 wait complete. 2024-09-20T19:40:50.429 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:50 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:50.429 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:50 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:50.429 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:50 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:50.429 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:50 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:40:50.430 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:50 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:50.430 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:50 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:40:50.430 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:50 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:50.430 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:50 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:50.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:50 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:50.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:50 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:50.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:50 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:50.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:50 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:40:50.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:50 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:50.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:50 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:40:50.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:50 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:50.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:50 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:50.805 DEBUG:teuthology.orchestra.run.smithi160:osd.3> sudo journalctl -f -n 0 -u ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@osd.3.service 2024-09-20T19:40:50.807 INFO:tasks.cephadm:Deploying osd.4 on smithi160 with /dev/vg_nvme/lv_3... 2024-09-20T19:40:50.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T19:40:51.699 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:51 smithi160 ceph-mon[35859]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:51.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:51 smithi114 ceph-mon[28260]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:52.070 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi160/config 2024-09-20T19:40:52.562 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:52 smithi160 ceph-mon[35859]: from='osd.3 [v2:172.21.15.160:6800/496816402,v1:172.21.15.160:6801/496816402]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T19:40:52.562 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:52 smithi160 ceph-mon[35859]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T19:40:52.562 INFO:journalctl@ceph.osd.3.smithi160.stdout:Sep 20 19:40:52 smithi160 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-3[41789]: 2024-09-20T19:40:52.321+0000 7f572d796640 -1 osd.3 0 log_to_monitors true 2024-09-20T19:40:52.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:52 smithi114 ceph-mon[28260]: from='osd.3 [v2:172.21.15.160:6800/496816402,v1:172.21.15.160:6801/496816402]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T19:40:52.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:52 smithi114 ceph-mon[28260]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T19:40:53.450 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:53 smithi160 ceph-mon[35859]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:53.450 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:53 smithi160 ceph-mon[35859]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:40:53.450 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:53 smithi160 ceph-mon[35859]: osdmap e25: 4 total, 3 up, 4 in 2024-09-20T19:40:53.450 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:53 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:53.450 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:53 smithi160 ceph-mon[35859]: from='osd.3 [v2:172.21.15.160:6800/496816402,v1:172.21.15.160:6801/496816402]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-20T19:40:53.450 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:53 smithi160 ceph-mon[35859]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-20T19:40:53.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:53 smithi114 ceph-mon[28260]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:53.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:53 smithi114 ceph-mon[28260]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T19:40:53.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:53 smithi114 ceph-mon[28260]: osdmap e25: 4 total, 3 up, 4 in 2024-09-20T19:40:53.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:53 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:53.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:53 smithi114 ceph-mon[28260]: from='osd.3 [v2:172.21.15.160:6800/496816402,v1:172.21.15.160:6801/496816402]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-20T19:40:53.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:53 smithi114 ceph-mon[28260]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-20T19:40:54.529 INFO:journalctl@ceph.osd.3.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-3[41789]: 2024-09-20T19:40:54.234+0000 7f5728ef6640 -1 osd.3 0 waiting for initial osdmap 2024-09-20T19:40:54.529 INFO:journalctl@ceph.osd.3.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-3[41789]: 2024-09-20T19:40:54.243+0000 7f5721528640 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:40:54.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-mon[35859]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-20T19:40:54.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-mon[35859]: osdmap e26: 4 total, 3 up, 4 in 2024-09-20T19:40:54.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:54.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:54.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-mon[35859]: from='osd.3 ' entity='osd.3' 2024-09-20T19:40:54.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:54.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:54.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:40:54.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:54.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:54.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:40:54.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:54.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-mon[35859]: osd.3 [v2:172.21.15.160:6800/496816402,v1:172.21.15.160:6801/496816402] boot 2024-09-20T19:40:54.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-mon[35859]: osdmap e27: 4 total, 4 up, 4 in 2024-09-20T19:40:54.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:54 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:54.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:54 smithi114 ceph-mon[28260]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-20T19:40:54.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:54 smithi114 ceph-mon[28260]: osdmap e26: 4 total, 3 up, 4 in 2024-09-20T19:40:54.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:54 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:54.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:54 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:54.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:54 smithi114 ceph-mon[28260]: from='osd.3 ' entity='osd.3' 2024-09-20T19:40:54.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:54 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:54.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:54 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:54.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:54 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:40:54.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:54 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:54.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:54 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:54.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:54 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:40:54.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:54 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:40:54.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:54 smithi114 ceph-mon[28260]: osd.3 [v2:172.21.15.160:6800/496816402,v1:172.21.15.160:6801/496816402] boot 2024-09-20T19:40:54.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:54 smithi114 ceph-mon[28260]: osdmap e27: 4 total, 4 up, 4 in 2024-09-20T19:40:54.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:54 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T19:40:55.630 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:40:55.656 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:55 smithi160 ceph-mon[35859]: purged_snaps scrub starts 2024-09-20T19:40:55.656 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:55 smithi160 ceph-mon[35859]: purged_snaps scrub ok 2024-09-20T19:40:55.656 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:55 smithi160 ceph-mon[35859]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:55.656 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:55 smithi160 ceph-mon[35859]: Detected new or changed devices on smithi160 2024-09-20T19:40:55.656 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:55 smithi160 ceph-mon[35859]: Adjusting osd_memory_target on smithi160 to 14912M 2024-09-20T19:40:55.656 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:55 smithi160 ceph-mon[35859]: osdmap e28: 4 total, 4 up, 4 in 2024-09-20T19:40:55.658 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi160:vg_nvme/lv_3 2024-09-20T19:40:55.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:55 smithi114 ceph-mon[28260]: purged_snaps scrub starts 2024-09-20T19:40:55.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:55 smithi114 ceph-mon[28260]: purged_snaps scrub ok 2024-09-20T19:40:55.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:55 smithi114 ceph-mon[28260]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-20T19:40:55.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:55 smithi114 ceph-mon[28260]: Detected new or changed devices on smithi160 2024-09-20T19:40:55.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:55 smithi114 ceph-mon[28260]: Adjusting osd_memory_target on smithi160 to 14912M 2024-09-20T19:40:55.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:55 smithi114 ceph-mon[28260]: osdmap e28: 4 total, 4 up, 4 in 2024-09-20T19:40:55.905 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi160/config 2024-09-20T19:40:56.364 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.329+0000 7fbb817f3640 1 -- 172.21.15.160:0/1710132689 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb7c102490 msgr2=0x7fbb7c104880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:40:56.364 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.329+0000 7fbb817f3640 1 --2- 172.21.15.160:0/1710132689 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb7c102490 0x7fbb7c104880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009880 tx=0x7fbb6c02a4a0 comp rx=0 tx=0).stop 2024-09-20T19:40:56.364 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.329+0000 7fbb817f3640 1 -- 172.21.15.160:0/1710132689 shutdown_connections 2024-09-20T19:40:56.364 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.329+0000 7fbb817f3640 1 --2- 172.21.15.160:0/1710132689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb7c104dc0 0x7fbb7c1071b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:40:56.364 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.329+0000 7fbb817f3640 1 --2- 172.21.15.160:0/1710132689 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb7c102490 0x7fbb7c104880 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:40:56.364 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.329+0000 7fbb817f3640 1 -- 172.21.15.160:0/1710132689 >> 172.21.15.160:0/1710132689 conn(0x7fbb7c0fbff0 msgr2=0x7fbb7c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:40:56.364 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.329+0000 7fbb817f3640 1 -- 172.21.15.160:0/1710132689 shutdown_connections 2024-09-20T19:40:56.364 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.329+0000 7fbb817f3640 1 -- 172.21.15.160:0/1710132689 wait complete. 2024-09-20T19:40:56.364 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.330+0000 7fbb817f3640 1 Processor -- start 2024-09-20T19:40:56.364 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.330+0000 7fbb817f3640 1 -- start start 2024-09-20T19:40:56.365 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.330+0000 7fbb817f3640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb7c102490 0x7fbb7c197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:40:56.365 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.330+0000 7fbb817f3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb7c104dc0 0x7fbb7c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:40:56.365 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.330+0000 7fbb817f3640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c10a800 con 0x7fbb7c104dc0 2024-09-20T19:40:56.365 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.330+0000 7fbb817f3640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c10a680 con 0x7fbb7c102490 2024-09-20T19:40:56.365 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.330+0000 7fbb7affd640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb7c102490 0x7fbb7c197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:40:56.365 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.330+0000 7fbb7affd640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb7c102490 0x7fbb7c197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:47732/0 (socket says 172.21.15.160:47732) 2024-09-20T19:40:56.365 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.330+0000 7fbb7affd640 1 -- 172.21.15.160:0/2054911880 learned_addr learned my addr 172.21.15.160:0/2054911880 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-09-20T19:40:56.365 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.330+0000 7fbb7a7fc640 1 --2- 172.21.15.160:0/2054911880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb7c104dc0 0x7fbb7c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:40:56.365 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.331+0000 7fbb7affd640 1 -- 172.21.15.160:0/2054911880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb7c104dc0 msgr2=0x7fbb7c197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:40:56.365 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.331+0000 7fbb7affd640 1 --2- 172.21.15.160:0/2054911880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb7c104dc0 0x7fbb7c197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:40:56.365 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.331+0000 7fbb7affd640 1 -- 172.21.15.160:0/2054911880 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7c1980e0 con 0x7fbb7c102490 2024-09-20T19:40:56.365 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.331+0000 7fbb7a7fc640 1 --2- 172.21.15.160:0/2054911880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb7c104dc0 0x7fbb7c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-20T19:40:56.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.331+0000 7fbb7affd640 1 --2- 172.21.15.160:0/2054911880 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb7c102490 0x7fbb7c197520 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c006010 tx=0x7fbb6c0048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:40:56.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.331+0000 7fbb63fff640 1 -- 172.21.15.160:0/2054911880 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fbb6c03f070 con 0x7fbb7c102490 2024-09-20T19:40:56.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.331+0000 7fbb63fff640 1 -- 172.21.15.160:0/2054911880 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbb6c0044d0 con 0x7fbb7c102490 2024-09-20T19:40:56.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.331+0000 7fbb817f3640 1 -- 172.21.15.160:0/2054911880 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb7c19cac0 con 0x7fbb7c102490 2024-09-20T19:40:56.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.331+0000 7fbb63fff640 1 -- 172.21.15.160:0/2054911880 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fbb6c035030 con 0x7fbb7c102490 2024-09-20T19:40:56.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.331+0000 7fbb817f3640 1 -- 172.21.15.160:0/2054911880 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb7c19d050 con 0x7fbb7c102490 2024-09-20T19:40:56.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.332+0000 7fbb817f3640 1 -- 172.21.15.160:0/2054911880 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb7c102870 con 0x7fbb7c102490 2024-09-20T19:40:56.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.332+0000 7fbb63fff640 1 -- 172.21.15.160:0/2054911880 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fbb6c042070 con 0x7fbb7c102490 2024-09-20T19:40:56.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.333+0000 7fbb63fff640 1 --2- 172.21.15.160:0/2054911880 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbb6406e2b0 0x7fbb64070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:40:56.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.333+0000 7fbb63fff640 1 -- 172.21.15.160:0/2054911880 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3803+0+0 (secure 0 0 0) 0x7fbb6c0ae890 con 0x7fbb7c102490 2024-09-20T19:40:56.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.333+0000 7fbb7a7fc640 1 --2- 172.21.15.160:0/2054911880 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbb6406e2b0 0x7fbb64070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:40:56.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.334+0000 7fbb7a7fc640 1 --2- 172.21.15.160:0/2054911880 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbb6406e2b0 0x7fbb64070770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbb70006fd0 tx=0x7fbb70008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:40:56.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.337+0000 7fbb63fff640 1 -- 172.21.15.160:0/2054911880 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fbb6c07c870 con 0x7fbb7c102490 2024-09-20T19:40:56.519 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:40:56.518+0000 7fbb817f3640 1 -- 172.21.15.160:0/2054911880 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fbb7c1057a0 con 0x7fbb6406e2b0 2024-09-20T19:40:56.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:56 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:40:56.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:56 smithi114 ceph-mon[28260]: osdmap e29: 4 total, 4 up, 4 in 2024-09-20T19:40:56.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:56 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:40:56.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:56 smithi160 ceph-mon[35859]: osdmap e29: 4 total, 4 up, 4 in 2024-09-20T19:40:57.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:57 smithi160 ceph-mon[35859]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:40:57.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:57 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:40:57.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:57 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:40:57.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:57 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:57.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:57 smithi114 ceph-mon[28260]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:40:57.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:57 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:40:57.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:57 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:40:57.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:57 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:40:58.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:58 smithi114 ceph-mon[28260]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:40:58.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:58 smithi114 ceph-mon[28260]: osdmap e30: 4 total, 4 up, 4 in 2024-09-20T19:40:58.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:58 smithi160 ceph-mon[35859]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:40:58.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:58 smithi160 ceph-mon[35859]: osdmap e30: 4 total, 4 up, 4 in 2024-09-20T19:40:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:59 smithi114 ceph-mon[28260]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:40:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:59 smithi114 ceph-mon[28260]: osdmap e31: 4 total, 4 up, 4 in 2024-09-20T19:40:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:59 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/3401913053' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "302c7425-1a84-44fc-a4c9-58a71a9cdc54"}]: dispatch 2024-09-20T19:40:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:59 smithi114 ceph-mon[28260]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "302c7425-1a84-44fc-a4c9-58a71a9cdc54"}]: dispatch 2024-09-20T19:40:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:59 smithi114 ceph-mon[28260]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "302c7425-1a84-44fc-a4c9-58a71a9cdc54"}]': finished 2024-09-20T19:40:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:59 smithi114 ceph-mon[28260]: osdmap e32: 5 total, 4 up, 5 in 2024-09-20T19:40:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:59 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:40:59.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:40:59 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/3240716245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:40:59.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:59 smithi160 ceph-mon[35859]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:40:59.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:59 smithi160 ceph-mon[35859]: osdmap e31: 4 total, 4 up, 4 in 2024-09-20T19:40:59.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:59 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/3401913053' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "302c7425-1a84-44fc-a4c9-58a71a9cdc54"}]: dispatch 2024-09-20T19:40:59.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:59 smithi160 ceph-mon[35859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "302c7425-1a84-44fc-a4c9-58a71a9cdc54"}]: dispatch 2024-09-20T19:40:59.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:59 smithi160 ceph-mon[35859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "302c7425-1a84-44fc-a4c9-58a71a9cdc54"}]': finished 2024-09-20T19:40:59.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:59 smithi160 ceph-mon[35859]: osdmap e32: 5 total, 4 up, 5 in 2024-09-20T19:40:59.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:59 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:40:59.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:40:59 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/3240716245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:41:01.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:01 smithi114 ceph-mon[28260]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:41:01.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:01 smithi160 ceph-mon[35859]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T19:41:03.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:03 smithi114 ceph-mon[28260]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-20T19:41:03.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:03 smithi160 ceph-mon[35859]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-20T19:41:05.505 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:05 smithi160 ceph-mon[35859]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-09-20T19:41:05.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:05 smithi114 ceph-mon[28260]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-09-20T19:41:07.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:07 smithi114 ceph-mon[28260]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:07.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:07 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T19:41:07.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:07 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:41:07.739 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:07 smithi160 ceph-mon[35859]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:07.740 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:07 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T19:41:07.740 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:07 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:41:08.636 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:08 smithi160 ceph-mon[35859]: Deploying daemon osd.4 on smithi160 2024-09-20T19:41:08.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:08 smithi114 ceph-mon[28260]: Deploying daemon osd.4 on smithi160 2024-09-20T19:41:09.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:09 smithi160 ceph-mon[35859]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:09.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:09 smithi114 ceph-mon[28260]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:11.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:11 smithi114 ceph-mon[28260]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:11.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:11 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:41:11.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:11 smithi160 ceph-mon[35859]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:11.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:11 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:41:13.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:13 smithi114 ceph-mon[28260]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:13.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:13 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:41:13.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:13 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:13.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:13 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:13.742 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:13 smithi160 ceph-mon[35859]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:13.742 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:13 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:41:13.742 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:13 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:13.742 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:13 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:15.642 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:15 smithi160 ceph-mon[35859]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:15.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:15 smithi114 ceph-mon[28260]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:17.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:17.364+0000 7fbb63fff640 1 -- 172.21.15.160:0/2054911880 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbb7c1057a0 con 0x7fbb6406e2b0 2024-09-20T19:41:17.367 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 4 on host 'smithi160' 2024-09-20T19:41:17.369 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:17.367+0000 7fbb817f3640 1 -- 172.21.15.160:0/2054911880 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbb6406e2b0 msgr2=0x7fbb64070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:17.369 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:17.367+0000 7fbb817f3640 1 --2- 172.21.15.160:0/2054911880 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbb6406e2b0 0x7fbb64070770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbb70006fd0 tx=0x7fbb70008040 comp rx=0 tx=0).stop 2024-09-20T19:41:17.369 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:17.367+0000 7fbb817f3640 1 -- 172.21.15.160:0/2054911880 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb7c102490 msgr2=0x7fbb7c197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:17.369 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:17.367+0000 7fbb817f3640 1 --2- 172.21.15.160:0/2054911880 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb7c102490 0x7fbb7c197520 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c006010 tx=0x7fbb6c0048f0 comp rx=0 tx=0).stop 2024-09-20T19:41:17.369 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:17.367+0000 7fbb817f3640 1 -- 172.21.15.160:0/2054911880 shutdown_connections 2024-09-20T19:41:17.370 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:17.367+0000 7fbb817f3640 1 --2- 172.21.15.160:0/2054911880 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbb6406e2b0 0x7fbb64070770 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:17.370 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:17.367+0000 7fbb817f3640 1 --2- 172.21.15.160:0/2054911880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb7c104dc0 0x7fbb7c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:17.370 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:17.367+0000 7fbb817f3640 1 --2- 172.21.15.160:0/2054911880 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb7c102490 0x7fbb7c197520 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:17.370 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:17.367+0000 7fbb817f3640 1 -- 172.21.15.160:0/2054911880 >> 172.21.15.160:0/2054911880 conn(0x7fbb7c0fbff0 msgr2=0x7fbb7c0fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:17.370 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:17.367+0000 7fbb817f3640 1 -- 172.21.15.160:0/2054911880 shutdown_connections 2024-09-20T19:41:17.370 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:17.367+0000 7fbb817f3640 1 -- 172.21.15.160:0/2054911880 wait complete. 2024-09-20T19:41:17.674 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:17 smithi160 ceph-mon[35859]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:17.674 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:17 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:17.674 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:17 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:17.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:17 smithi114 ceph-mon[28260]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:17.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:17 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:17.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:17 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:17.985 DEBUG:teuthology.orchestra.run.smithi160:osd.4> sudo journalctl -f -n 0 -u ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@osd.4.service 2024-09-20T19:41:17.988 INFO:tasks.cephadm:Deploying osd.5 on smithi160 with /dev/vg_nvme/lv_2... 2024-09-20T19:41:17.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T19:41:18.282 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi160/config 2024-09-20T19:41:19.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:19 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:19.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:19 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:19.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:19 smithi160 ceph-mon[35859]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:19.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:19 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:19.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:19 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:19.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:19 smithi114 ceph-mon[28260]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:19.725 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 20 19:41:19 smithi160 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-4[48666]: 2024-09-20T19:41:19.587+0000 7fe24caa2640 -1 osd.4 0 log_to_monitors true 2024-09-20T19:41:20.379 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:20 smithi160 ceph-mon[35859]: from='osd.4 [v2:172.21.15.160:6808/3464010154,v1:172.21.15.160:6809/3464010154]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T19:41:20.379 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:20 smithi160 ceph-mon[35859]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T19:41:20.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:20 smithi114 ceph-mon[28260]: from='osd.4 [v2:172.21.15.160:6808/3464010154,v1:172.21.15.160:6809/3464010154]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T19:41:20.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:20 smithi114 ceph-mon[28260]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T19:41:21.418 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 20 19:41:21 smithi160 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-4[48666]: 2024-09-20T19:41:21.088+0000 7fe248202640 -1 osd.4 0 waiting for initial osdmap 2024-09-20T19:41:21.419 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 20 19:41:21 smithi160 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-4[48666]: 2024-09-20T19:41:21.094+0000 7fe24483c640 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:41:21.419 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:21 smithi160 ceph-mon[35859]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:41:21.419 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:21 smithi160 ceph-mon[35859]: osdmap e33: 5 total, 4 up, 5 in 2024-09-20T19:41:21.419 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:21 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:41:21.419 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:21 smithi160 ceph-mon[35859]: from='osd.4 [v2:172.21.15.160:6808/3464010154,v1:172.21.15.160:6809/3464010154]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-20T19:41:21.419 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:21 smithi160 ceph-mon[35859]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-20T19:41:21.419 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:21 smithi160 ceph-mon[35859]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:21.421 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-20T19:41:21.449 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi160:vg_nvme/lv_2 2024-09-20T19:41:21.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:21 smithi114 ceph-mon[28260]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T19:41:21.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:21 smithi114 ceph-mon[28260]: osdmap e33: 5 total, 4 up, 5 in 2024-09-20T19:41:21.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:21 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:41:21.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:21 smithi114 ceph-mon[28260]: from='osd.4 [v2:172.21.15.160:6808/3464010154,v1:172.21.15.160:6809/3464010154]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-20T19:41:21.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:21 smithi114 ceph-mon[28260]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-20T19:41:21.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:21 smithi114 ceph-mon[28260]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-20T19:41:22.002 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi160/config 2024-09-20T19:41:22.416 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:22 smithi160 ceph-mon[35859]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-20T19:41:22.416 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:22 smithi160 ceph-mon[35859]: osdmap e34: 5 total, 4 up, 5 in 2024-09-20T19:41:22.416 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:22 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:41:22.416 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:22 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:41:22.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:22 smithi114 ceph-mon[28260]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-20T19:41:22.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:22 smithi114 ceph-mon[28260]: osdmap e34: 5 total, 4 up, 5 in 2024-09-20T19:41:22.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:22 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:41:22.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:22 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:41:22.524 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.500+0000 7f00d0391640 1 -- 172.21.15.160:0/3875571502 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f00c8102400 msgr2=0x7f00c81047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:22.525 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.500+0000 7f00d0391640 1 --2- 172.21.15.160:0/3875571502 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f00c8102400 0x7f00c81047f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009880 tx=0x7f00bc02a4a0 comp rx=0 tx=0).stop 2024-09-20T19:41:22.525 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.501+0000 7f00d0391640 1 -- 172.21.15.160:0/3875571502 shutdown_connections 2024-09-20T19:41:22.525 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.501+0000 7f00d0391640 1 --2- 172.21.15.160:0/3875571502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c8104d30 0x7f00c8107120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:22.525 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.501+0000 7f00d0391640 1 --2- 172.21.15.160:0/3875571502 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f00c8102400 0x7f00c81047f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:22.525 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.501+0000 7f00d0391640 1 -- 172.21.15.160:0/3875571502 >> 172.21.15.160:0/3875571502 conn(0x7f00c80fc020 msgr2=0x7f00c80fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:22.525 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.501+0000 7f00d0391640 1 -- 172.21.15.160:0/3875571502 shutdown_connections 2024-09-20T19:41:22.525 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.501+0000 7f00d0391640 1 -- 172.21.15.160:0/3875571502 wait complete. 2024-09-20T19:41:22.525 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.501+0000 7f00d0391640 1 Processor -- start 2024-09-20T19:41:22.525 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.501+0000 7f00d0391640 1 -- start start 2024-09-20T19:41:22.525 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.501+0000 7f00d0391640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c8102400 0x7f00c8197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:22.525 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.501+0000 7f00d0391640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f00c8104d30 0x7f00c8197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:22.525 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.501+0000 7f00d0391640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f00c810a840 con 0x7f00c8102400 2024-09-20T19:41:22.525 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.501+0000 7f00d0391640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f00c810a6c0 con 0x7f00c8104d30 2024-09-20T19:41:22.526 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.502+0000 7f00cd92c640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f00c8104d30 0x7f00c8197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:22.526 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.502+0000 7f00cd92c640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f00c8104d30 0x7f00c8197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:48086/0 (socket says 172.21.15.160:48086) 2024-09-20T19:41:22.526 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.502+0000 7f00cd92c640 1 -- 172.21.15.160:0/2987869152 learned_addr learned my addr 172.21.15.160:0/2987869152 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-09-20T19:41:22.526 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.502+0000 7f00ce12d640 1 --2- 172.21.15.160:0/2987869152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c8102400 0x7f00c8197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:22.526 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.502+0000 7f00cd92c640 1 -- 172.21.15.160:0/2987869152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c8102400 msgr2=0x7f00c8197500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:22.526 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.502+0000 7f00cd92c640 1 --2- 172.21.15.160:0/2987869152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c8102400 0x7f00c8197500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:22.526 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.502+0000 7f00cd92c640 1 -- 172.21.15.160:0/2987869152 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c81980c0 con 0x7f00c8104d30 2024-09-20T19:41:22.526 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.502+0000 7f00cd92c640 1 --2- 172.21.15.160:0/2987869152 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f00c8104d30 0x7f00c8197a40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f00c400b3d0 tx=0x7f00c400b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:22.526 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.502+0000 7f00bb7fe640 1 -- 172.21.15.160:0/2987869152 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f00c4015020 con 0x7f00c8104d30 2024-09-20T19:41:22.526 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.502+0000 7f00bb7fe640 1 -- 172.21.15.160:0/2987869152 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f00c4011af0 con 0x7f00c8104d30 2024-09-20T19:41:22.527 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.502+0000 7f00bb7fe640 1 -- 172.21.15.160:0/2987869152 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f00c4010d30 con 0x7f00c8104d30 2024-09-20T19:41:22.527 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.502+0000 7f00d0391640 1 -- 172.21.15.160:0/2987869152 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c819cb00 con 0x7f00c8104d30 2024-09-20T19:41:22.527 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.502+0000 7f00d0391640 1 -- 172.21.15.160:0/2987869152 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c819d110 con 0x7f00c8104d30 2024-09-20T19:41:22.527 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.503+0000 7f00d0391640 1 -- 172.21.15.160:0/2987869152 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c81027e0 con 0x7f00c8104d30 2024-09-20T19:41:22.527 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.504+0000 7f00bb7fe640 1 -- 172.21.15.160:0/2987869152 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f00c4018020 con 0x7f00c8104d30 2024-09-20T19:41:22.527 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.505+0000 7f00bb7fe640 1 --2- 172.21.15.160:0/2987869152 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f00b406e2b0 0x7f00b4070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:22.527 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.505+0000 7f00bb7fe640 1 -- 172.21.15.160:0/2987869152 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4206+0+0 (secure 0 0 0) 0x7f00c408d6c0 con 0x7f00c8104d30 2024-09-20T19:41:22.527 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.505+0000 7f00ce12d640 1 --2- 172.21.15.160:0/2987869152 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f00b406e2b0 0x7f00b4070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:22.527 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.505+0000 7f00ce12d640 1 --2- 172.21.15.160:0/2987869152 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f00b406e2b0 0x7f00b4070770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f00bc000c00 tx=0x7f00bc004f30 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:22.527 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.509+0000 7f00bb7fe640 1 -- 172.21.15.160:0/2987869152 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f00c401f020 con 0x7f00c8104d30 2024-09-20T19:41:22.701 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:22.698+0000 7f00d0391640 1 -- 172.21.15.160:0/2987869152 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f00c81056b0 con 0x7f00b406e2b0 2024-09-20T19:41:23.404 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:23 smithi160 ceph-mon[35859]: purged_snaps scrub starts 2024-09-20T19:41:23.404 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:23 smithi160 ceph-mon[35859]: purged_snaps scrub ok 2024-09-20T19:41:23.404 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:23 smithi160 ceph-mon[35859]: osd.4 [v2:172.21.15.160:6808/3464010154,v1:172.21.15.160:6809/3464010154] boot 2024-09-20T19:41:23.405 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:23 smithi160 ceph-mon[35859]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T19:41:23.405 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:23 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:41:23.405 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:23 smithi160 ceph-mon[35859]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:23.405 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:23 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:41:23.405 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:23 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:41:23.405 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:23 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:41:23.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:23 smithi114 ceph-mon[28260]: purged_snaps scrub starts 2024-09-20T19:41:23.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:23 smithi114 ceph-mon[28260]: purged_snaps scrub ok 2024-09-20T19:41:23.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:23 smithi114 ceph-mon[28260]: osd.4 [v2:172.21.15.160:6808/3464010154,v1:172.21.15.160:6809/3464010154] boot 2024-09-20T19:41:23.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:23 smithi114 ceph-mon[28260]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T19:41:23.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:23 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T19:41:23.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:23 smithi114 ceph-mon[28260]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:23.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:23 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T19:41:23.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:23 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T19:41:23.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:23 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:41:24.310 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:41:24.310 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: osdmap e36: 5 total, 5 up, 5 in 2024-09-20T19:41:24.310 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: Detected new or changed devices on smithi160 2024-09-20T19:41:24.310 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:24.310 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:24.310 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:41:24.310 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:41:24.310 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: Adjusting osd_memory_target on smithi160 to 7456M 2024-09-20T19:41:24.311 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:24.311 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:41:24.311 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:41:24.311 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:24.311 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:41:24.311 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:41:24.311 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:41:24.311 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:24 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:24.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:41:24.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: osdmap e36: 5 total, 5 up, 5 in 2024-09-20T19:41:24.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: Detected new or changed devices on smithi160 2024-09-20T19:41:24.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:24.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:24.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:41:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:41:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: Adjusting osd_memory_target on smithi160 to 7456M 2024-09-20T19:41:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:41:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:41:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:41:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:41:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:41:24.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:24 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:25.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:25 smithi160 ceph-mon[35859]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:25.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:25 smithi114 ceph-mon[28260]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:26.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:26 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/4175681795' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a645a156-dc73-4033-adde-4d2380a9c000"}]: dispatch 2024-09-20T19:41:26.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:26 smithi114 ceph-mon[28260]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a645a156-dc73-4033-adde-4d2380a9c000"}]: dispatch 2024-09-20T19:41:26.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:26 smithi114 ceph-mon[28260]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a645a156-dc73-4033-adde-4d2380a9c000"}]': finished 2024-09-20T19:41:26.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:26 smithi114 ceph-mon[28260]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T19:41:26.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:26 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:41:26.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:26 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/3253867983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:41:26.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:26 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/4175681795' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a645a156-dc73-4033-adde-4d2380a9c000"}]: dispatch 2024-09-20T19:41:26.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:26 smithi160 ceph-mon[35859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a645a156-dc73-4033-adde-4d2380a9c000"}]: dispatch 2024-09-20T19:41:26.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:26 smithi160 ceph-mon[35859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a645a156-dc73-4033-adde-4d2380a9c000"}]': finished 2024-09-20T19:41:26.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:26 smithi160 ceph-mon[35859]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T19:41:26.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:26 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:41:26.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:26 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/3253867983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T19:41:27.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:27 smithi114 ceph-mon[28260]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:27.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:27 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:41:27.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:27 smithi160 ceph-mon[35859]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:27.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:27 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:41:29.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:29 smithi114 ceph-mon[28260]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:29.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:29 smithi160 ceph-mon[35859]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:31.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:31 smithi160 ceph-mon[35859]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:31.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:31 smithi114 ceph-mon[28260]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:33.663 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:33 smithi160 ceph-mon[35859]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:33.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:33 smithi114 ceph-mon[28260]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:34.466 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:34 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T19:41:34.466 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:34 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:41:34.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:34 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T19:41:34.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:34 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:41:35.522 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:35 smithi160 ceph-mon[35859]: Deploying daemon osd.5 on smithi160 2024-09-20T19:41:35.522 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:35 smithi160 ceph-mon[35859]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:35.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:35 smithi114 ceph-mon[28260]: Deploying daemon osd.5 on smithi160 2024-09-20T19:41:35.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:35 smithi114 ceph-mon[28260]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:37.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:37 smithi160 ceph-mon[35859]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:37.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:37 smithi114 ceph-mon[28260]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:39.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:39 smithi114 ceph-mon[28260]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:39.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:39 smithi160 ceph-mon[35859]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:40.434 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:40 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:41:40.435 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:40 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:40.435 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:40 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:40.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:40 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:41:40.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:40 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:40.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:40 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:41.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:41 smithi114 ceph-mon[28260]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:41.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:41 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:41:41.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:41 smithi160 ceph-mon[35859]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:41.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:41 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:41:43.022 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:43.020+0000 7f00bb7fe640 1 -- 172.21.15.160:0/2987869152 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f00c81056b0 con 0x7f00b406e2b0 2024-09-20T19:41:43.022 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 5 on host 'smithi160' 2024-09-20T19:41:43.025 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:43.023+0000 7f00d0391640 1 -- 172.21.15.160:0/2987869152 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f00b406e2b0 msgr2=0x7f00b4070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:43.025 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:43.023+0000 7f00d0391640 1 --2- 172.21.15.160:0/2987869152 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f00b406e2b0 0x7f00b4070770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f00bc000c00 tx=0x7f00bc004f30 comp rx=0 tx=0).stop 2024-09-20T19:41:43.025 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:43.023+0000 7f00d0391640 1 -- 172.21.15.160:0/2987869152 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f00c8104d30 msgr2=0x7f00c8197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:43.025 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:43.023+0000 7f00d0391640 1 --2- 172.21.15.160:0/2987869152 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f00c8104d30 0x7f00c8197a40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f00c400b3d0 tx=0x7f00c400b890 comp rx=0 tx=0).stop 2024-09-20T19:41:43.025 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:43.023+0000 7f00d0391640 1 -- 172.21.15.160:0/2987869152 shutdown_connections 2024-09-20T19:41:43.025 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:43.023+0000 7f00d0391640 1 --2- 172.21.15.160:0/2987869152 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f00b406e2b0 0x7f00b4070770 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:43.025 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:43.023+0000 7f00d0391640 1 --2- 172.21.15.160:0/2987869152 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f00c8104d30 0x7f00c8197a40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:43.025 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:43.023+0000 7f00d0391640 1 --2- 172.21.15.160:0/2987869152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00c8102400 0x7f00c8197500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:43.025 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:43.023+0000 7f00d0391640 1 -- 172.21.15.160:0/2987869152 >> 172.21.15.160:0/2987869152 conn(0x7f00c80fc020 msgr2=0x7f00c80fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:43.025 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:43.023+0000 7f00d0391640 1 -- 172.21.15.160:0/2987869152 shutdown_connections 2024-09-20T19:41:43.025 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:43.023+0000 7f00d0391640 1 -- 172.21.15.160:0/2987869152 wait complete. 2024-09-20T19:41:43.632 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:43 smithi160 ceph-mon[35859]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:43.632 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:43 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:43.632 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:43 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:43.634 DEBUG:teuthology.orchestra.run.smithi160:osd.5> sudo journalctl -f -n 0 -u ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@osd.5.service 2024-09-20T19:41:43.637 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-09-20T19:41:43.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-20T19:41:43.665 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:43 smithi114 ceph-mon[28260]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:43.666 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:43 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:43.666 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:43 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:43.903 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:41:44.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.345+0000 7feaf3569640 1 -- 172.21.15.114:0/148178686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaec100840 msgr2=0x7feaec100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:44.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.345+0000 7feaf3569640 1 --2- 172.21.15.114:0/148178686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaec100840 0x7feaec100c40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7feae8009880 tx=0x7feae802a4a0 comp rx=0 tx=0).stop 2024-09-20T19:41:44.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.345+0000 7feaf3569640 1 -- 172.21.15.114:0/148178686 shutdown_connections 2024-09-20T19:41:44.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.345+0000 7feaf3569640 1 --2- 172.21.15.114:0/148178686 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feaec101a40 0x7feaec101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:44.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.345+0000 7feaf3569640 1 --2- 172.21.15.114:0/148178686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaec100840 0x7feaec100c40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:44.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.345+0000 7feaf3569640 1 -- 172.21.15.114:0/148178686 >> 172.21.15.114:0/148178686 conn(0x7feaec0fbff0 msgr2=0x7feaec0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:44.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.345+0000 7feaf3569640 1 -- 172.21.15.114:0/148178686 shutdown_connections 2024-09-20T19:41:44.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.345+0000 7feaf3569640 1 -- 172.21.15.114:0/148178686 wait complete. 2024-09-20T19:41:44.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.345+0000 7feaf3569640 1 Processor -- start 2024-09-20T19:41:44.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.345+0000 7feaf3569640 1 -- start start 2024-09-20T19:41:44.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.346+0000 7feaf3569640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feaec100840 0x7feaec06a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:44.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.346+0000 7feaf3569640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaec101a40 0x7feaec19c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:44.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.346+0000 7feaf3569640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feaec109660 con 0x7feaec101a40 2024-09-20T19:41:44.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.346+0000 7feaf3569640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7feaec1094e0 con 0x7feaec100840 2024-09-20T19:41:44.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.346+0000 7feaf0b04640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaec101a40 0x7feaec19c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:44.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.346+0000 7feaf0b04640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaec101a40 0x7feaec19c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36766/0 (socket says 172.21.15.114:36766) 2024-09-20T19:41:44.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.346+0000 7feaf0b04640 1 -- 172.21.15.114:0/2098131820 learned_addr learned my addr 172.21.15.114:0/2098131820 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:41:44.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.346+0000 7feaf0b04640 1 -- 172.21.15.114:0/2098131820 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feaec100840 msgr2=0x7feaec06a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:44.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.346+0000 7feaf0b04640 1 --2- 172.21.15.114:0/2098131820 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feaec100840 0x7feaec06a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:44.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.346+0000 7feaf0b04640 1 -- 172.21.15.114:0/2098131820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaec19c7a0 con 0x7feaec101a40 2024-09-20T19:41:44.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.346+0000 7feaf0b04640 1 --2- 172.21.15.114:0/2098131820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaec101a40 0x7feaec19c1b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7feadc00d8d0 tx=0x7feadc00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:44.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.347+0000 7feae27fc640 1 -- 172.21.15.114:0/2098131820 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7feadc017070 con 0x7feaec101a40 2024-09-20T19:41:44.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.347+0000 7feaf3569640 1 -- 172.21.15.114:0/2098131820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaec19ca90 con 0x7feaec101a40 2024-09-20T19:41:44.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.347+0000 7feae27fc640 1 -- 172.21.15.114:0/2098131820 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7feadc010460 con 0x7feaec101a40 2024-09-20T19:41:44.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.347+0000 7feae27fc640 1 -- 172.21.15.114:0/2098131820 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7feadc00f5f0 con 0x7feaec101a40 2024-09-20T19:41:44.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.347+0000 7feaf3569640 1 -- 172.21.15.114:0/2098131820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaec19d020 con 0x7feaec101a40 2024-09-20T19:41:44.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.348+0000 7feaf3569640 1 -- 172.21.15.114:0/2098131820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaec100c40 con 0x7feaec101a40 2024-09-20T19:41:44.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.349+0000 7feae27fc640 1 -- 172.21.15.114:0/2098131820 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7feadc009a30 con 0x7feaec101a40 2024-09-20T19:41:44.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.349+0000 7feae27fc640 1 --2- 172.21.15.114:0/2098131820 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fead806e310 0x7fead80707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:44.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.349+0000 7feae27fc640 1 -- 172.21.15.114:0/2098131820 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4321+0+0 (secure 0 0 0) 0x7feadc08db90 con 0x7feaec101a40 2024-09-20T19:41:44.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.349+0000 7feaf1305640 1 --2- 172.21.15.114:0/2098131820 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fead806e310 0x7fead80707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:44.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.350+0000 7feaf1305640 1 --2- 172.21.15.114:0/2098131820 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fead806e310 0x7fead80707d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feae80096c0 tx=0x7feae8000bc0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:44.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.354+0000 7feae27fc640 1 -- 172.21.15.114:0/2098131820 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7feadc05b950 con 0x7feaec101a40 2024-09-20T19:41:44.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.527+0000 7feaf3569640 1 -- 172.21.15.114:0/2098131820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7feaec062df0 con 0x7feaec101a40 2024-09-20T19:41:44.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.527+0000 7feae27fc640 1 -- 172.21.15.114:0/2098131820 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v37) ==== 74+0+130 (secure 0 0 0) 0x7feadc05f600 con 0x7feaec101a40 2024-09-20T19:41:44.529 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:41:44.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.530+0000 7feaf3569640 1 -- 172.21.15.114:0/2098131820 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fead806e310 msgr2=0x7fead80707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:44.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.530+0000 7feaf3569640 1 --2- 172.21.15.114:0/2098131820 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fead806e310 0x7fead80707d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feae80096c0 tx=0x7feae8000bc0 comp rx=0 tx=0).stop 2024-09-20T19:41:44.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.530+0000 7feaf3569640 1 -- 172.21.15.114:0/2098131820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaec101a40 msgr2=0x7feaec19c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:44.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.530+0000 7feaf3569640 1 --2- 172.21.15.114:0/2098131820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaec101a40 0x7feaec19c1b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7feadc00d8d0 tx=0x7feadc00dd90 comp rx=0 tx=0).stop 2024-09-20T19:41:44.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.530+0000 7feaf3569640 1 -- 172.21.15.114:0/2098131820 shutdown_connections 2024-09-20T19:41:44.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.530+0000 7feaf3569640 1 --2- 172.21.15.114:0/2098131820 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fead806e310 0x7fead80707d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:44.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.530+0000 7feaf3569640 1 --2- 172.21.15.114:0/2098131820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaec101a40 0x7feaec19c1b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:44.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.530+0000 7feaf3569640 1 --2- 172.21.15.114:0/2098131820 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feaec100840 0x7feaec06a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:44.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.530+0000 7feaf3569640 1 -- 172.21.15.114:0/2098131820 >> 172.21.15.114:0/2098131820 conn(0x7feaec0fbff0 msgr2=0x7feaec104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:44.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.530+0000 7feaf3569640 1 -- 172.21.15.114:0/2098131820 shutdown_connections 2024-09-20T19:41:44.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:44.530+0000 7feaf3569640 1 -- 172.21.15.114:0/2098131820 wait complete. 2024-09-20T19:41:44.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:44 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:44.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:44 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:44.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:44 smithi160 ceph-mon[35859]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:44.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:44 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/2098131820' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T19:41:44.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:44 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:44.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:44 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:44.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:44 smithi114 ceph-mon[28260]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:44.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:44 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/2098131820' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T19:41:45.372 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":37,"num_osds":6,"num_up_osds":5,"osd_up_since":1726861282,"num_in_osds":6,"osd_in_since":1726861285,"num_remapped_pgs":0} 2024-09-20T19:41:45.750 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 20 19:41:45 smithi160 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-5[55145]: 2024-09-20T19:41:45.659+0000 7f8bba771640 -1 osd.5 0 log_to_monitors true 2024-09-20T19:41:46.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:45 smithi160 ceph-mon[35859]: from='osd.5 [v2:172.21.15.160:6816/3716955701,v1:172.21.15.160:6817/3716955701]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T19:41:46.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:45 smithi160 ceph-mon[35859]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T19:41:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:45 smithi114 ceph-mon[28260]: from='osd.5 [v2:172.21.15.160:6816/3716955701,v1:172.21.15.160:6817/3716955701]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T19:41:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:45 smithi114 ceph-mon[28260]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T19:41:46.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-20T19:41:46.626 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:41:47.055 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:46 smithi114 ceph-mon[28260]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T19:41:47.056 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:46 smithi114 ceph-mon[28260]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T19:41:47.056 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:46 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:41:47.056 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:46 smithi114 ceph-mon[28260]: from='osd.5 [v2:172.21.15.160:6816/3716955701,v1:172.21.15.160:6817/3716955701]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-20T19:41:47.056 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:46 smithi114 ceph-mon[28260]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-20T19:41:47.056 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:46 smithi114 ceph-mon[28260]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:47.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.068+0000 7fd8917a9640 1 -- 172.21.15.114:0/2601514800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd88c101a40 msgr2=0x7fd88c101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:47.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.068+0000 7fd8917a9640 1 --2- 172.21.15.114:0/2601514800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd88c101a40 0x7fd88c101ec0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd874009880 tx=0x7fd87402a4a0 comp rx=0 tx=0).stop 2024-09-20T19:41:47.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.068+0000 7fd8917a9640 1 -- 172.21.15.114:0/2601514800 shutdown_connections 2024-09-20T19:41:47.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.068+0000 7fd8917a9640 1 --2- 172.21.15.114:0/2601514800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd88c101a40 0x7fd88c101ec0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:47.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.068+0000 7fd8917a9640 1 --2- 172.21.15.114:0/2601514800 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd88c100840 0x7fd88c100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:47.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.068+0000 7fd8917a9640 1 -- 172.21.15.114:0/2601514800 >> 172.21.15.114:0/2601514800 conn(0x7fd88c0fbff0 msgr2=0x7fd88c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:47.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.069+0000 7fd8917a9640 1 -- 172.21.15.114:0/2601514800 shutdown_connections 2024-09-20T19:41:47.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.069+0000 7fd8917a9640 1 -- 172.21.15.114:0/2601514800 wait complete. 2024-09-20T19:41:47.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.069+0000 7fd8917a9640 1 Processor -- start 2024-09-20T19:41:47.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.069+0000 7fd8917a9640 1 -- start start 2024-09-20T19:41:47.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.069+0000 7fd8917a9640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd88c100840 0x7fd88c06a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:47.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.069+0000 7fd8917a9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd88c101a40 0x7fd88c19c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:47.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.069+0000 7fd8917a9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c109680 con 0x7fd88c101a40 2024-09-20T19:41:47.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.069+0000 7fd8917a9640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c109500 con 0x7fd88c100840 2024-09-20T19:41:47.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.069+0000 7fd883fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd88c101a40 0x7fd88c19c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:47.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.069+0000 7fd883fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd88c101a40 0x7fd88c19c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45194/0 (socket says 172.21.15.114:45194) 2024-09-20T19:41:47.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.069+0000 7fd883fff640 1 -- 172.21.15.114:0/1345884142 learned_addr learned my addr 172.21.15.114:0/1345884142 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:41:47.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.069+0000 7fd88affd640 1 --2- 172.21.15.114:0/1345884142 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd88c100840 0x7fd88c06a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:47.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.070+0000 7fd883fff640 1 -- 172.21.15.114:0/1345884142 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd88c100840 msgr2=0x7fd88c06a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:47.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.070+0000 7fd883fff640 1 --2- 172.21.15.114:0/1345884142 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd88c100840 0x7fd88c06a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:47.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.070+0000 7fd883fff640 1 -- 172.21.15.114:0/1345884142 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88c19c7e0 con 0x7fd88c101a40 2024-09-20T19:41:47.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.070+0000 7fd88affd640 1 --2- 172.21.15.114:0/1345884142 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd88c100840 0x7fd88c06a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-20T19:41:47.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.070+0000 7fd883fff640 1 --2- 172.21.15.114:0/1345884142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd88c101a40 0x7fd88c19c220 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd874006010 tx=0x7fd874004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:47.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.070+0000 7fd888ff9640 1 -- 172.21.15.114:0/1345884142 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fd87403f070 con 0x7fd88c101a40 2024-09-20T19:41:47.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.070+0000 7fd888ff9640 1 -- 172.21.15.114:0/1345884142 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd8740049d0 con 0x7fd88c101a40 2024-09-20T19:41:47.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.070+0000 7fd8917a9640 1 -- 172.21.15.114:0/1345884142 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd88c19ca70 con 0x7fd88c101a40 2024-09-20T19:41:47.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.070+0000 7fd888ff9640 1 -- 172.21.15.114:0/1345884142 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fd874035030 con 0x7fd88c101a40 2024-09-20T19:41:47.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.070+0000 7fd8917a9640 1 -- 172.21.15.114:0/1345884142 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd88c19cfa0 con 0x7fd88c101a40 2024-09-20T19:41:47.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.071+0000 7fd8917a9640 1 -- 172.21.15.114:0/1345884142 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd88c100c40 con 0x7fd88c101a40 2024-09-20T19:41:47.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.072+0000 7fd888ff9640 1 -- 172.21.15.114:0/1345884142 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fd874042070 con 0x7fd88c101a40 2024-09-20T19:41:47.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.072+0000 7fd888ff9640 1 --2- 172.21.15.114:0/1345884142 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd87806e310 0x7fd8780707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:47.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.072+0000 7fd888ff9640 1 -- 172.21.15.114:0/1345884142 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4358+0+0 (secure 0 0 0) 0x7fd8740ae820 con 0x7fd88c101a40 2024-09-20T19:41:47.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.072+0000 7fd88affd640 1 --2- 172.21.15.114:0/1345884142 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd87806e310 0x7fd8780707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:47.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.073+0000 7fd88affd640 1 --2- 172.21.15.114:0/1345884142 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd87806e310 0x7fd8780707d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd88c1018a0 tx=0x7fd87c005f90 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:47.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.077+0000 7fd888ff9640 1 -- 172.21.15.114:0/1345884142 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fd874080270 con 0x7fd88c101a40 2024-09-20T19:41:47.111 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:46 smithi160 ceph-mon[35859]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T19:41:47.111 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:46 smithi160 ceph-mon[35859]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T19:41:47.111 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:46 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:41:47.111 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:46 smithi160 ceph-mon[35859]: from='osd.5 [v2:172.21.15.160:6816/3716955701,v1:172.21.15.160:6817/3716955701]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-20T19:41:47.111 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:46 smithi160 ceph-mon[35859]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-20T19:41:47.112 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:46 smithi160 ceph-mon[35859]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:47.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.251+0000 7fd8917a9640 1 -- 172.21.15.114:0/1345884142 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fd88c062df0 con 0x7fd88c101a40 2024-09-20T19:41:47.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.252+0000 7fd888ff9640 1 -- 172.21.15.114:0/1345884142 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) ==== 74+0+130 (secure 0 0 0) 0x7fd874084aa0 con 0x7fd88c101a40 2024-09-20T19:41:47.254 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:41:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.254+0000 7fd8917a9640 1 -- 172.21.15.114:0/1345884142 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd87806e310 msgr2=0x7fd8780707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.254+0000 7fd8917a9640 1 --2- 172.21.15.114:0/1345884142 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd87806e310 0x7fd8780707d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd88c1018a0 tx=0x7fd87c005f90 comp rx=0 tx=0).stop 2024-09-20T19:41:47.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.254+0000 7fd8917a9640 1 -- 172.21.15.114:0/1345884142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd88c101a40 msgr2=0x7fd88c19c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:47.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.254+0000 7fd8917a9640 1 --2- 172.21.15.114:0/1345884142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd88c101a40 0x7fd88c19c220 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd874006010 tx=0x7fd874004570 comp rx=0 tx=0).stop 2024-09-20T19:41:47.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.254+0000 7fd8917a9640 1 -- 172.21.15.114:0/1345884142 shutdown_connections 2024-09-20T19:41:47.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.254+0000 7fd8917a9640 1 --2- 172.21.15.114:0/1345884142 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd87806e310 0x7fd8780707d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:47.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.254+0000 7fd8917a9640 1 --2- 172.21.15.114:0/1345884142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd88c101a40 0x7fd88c19c220 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:47.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.254+0000 7fd8917a9640 1 --2- 172.21.15.114:0/1345884142 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd88c100840 0x7fd88c06a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:47.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.254+0000 7fd8917a9640 1 -- 172.21.15.114:0/1345884142 >> 172.21.15.114:0/1345884142 conn(0x7fd88c0fbff0 msgr2=0x7fd88c104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:47.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.254+0000 7fd8917a9640 1 -- 172.21.15.114:0/1345884142 shutdown_connections 2024-09-20T19:41:47.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:47.254+0000 7fd8917a9640 1 -- 172.21.15.114:0/1345884142 wait complete. 2024-09-20T19:41:48.003 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":39,"num_osds":6,"num_up_osds":5,"osd_up_since":1726861282,"num_in_osds":6,"osd_in_since":1726861285,"num_remapped_pgs":0} 2024-09-20T19:41:48.029 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-5[55145]: 2024-09-20T19:41:47.623+0000 7f8bb5ed1640 -1 osd.5 0 waiting for initial osdmap 2024-09-20T19:41:48.029 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-5[55145]: 2024-09-20T19:41:47.630+0000 7f8bae503640 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T19:41:48.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-20T19:41:48.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: osdmap e39: 6 total, 5 up, 6 in 2024-09-20T19:41:48.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:41:48.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:41:48.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: Detected new or changed devices on smithi160 2024-09-20T19:41:48.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:48.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:48.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:41:48.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:41:48.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:41:48.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: Adjusting osd_memory_target on smithi160 to 4970M 2024-09-20T19:41:48.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:48.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:41:48.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:41:48.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:48.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1345884142' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T19:41:48.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:41:48.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:41:48.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:41:48.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:48.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:47 smithi160 ceph-mon[35859]: from='osd.5 ' entity='osd.5' 2024-09-20T19:41:48.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-20T19:41:48.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: osdmap e39: 6 total, 5 up, 6 in 2024-09-20T19:41:48.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:41:48.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:41:48.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: Detected new or changed devices on smithi160 2024-09-20T19:41:48.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:48.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:48.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:41:48.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:41:48.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T19:41:48.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: Adjusting osd_memory_target on smithi160 to 4970M 2024-09-20T19:41:48.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:48.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:41:48.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:41:48.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:48.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1345884142' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T19:41:48.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:41:48.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:41:48.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:41:48.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:41:48.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:47 smithi114 ceph-mon[28260]: from='osd.5 ' entity='osd.5' 2024-09-20T19:41:49.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-20T19:41:49.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:48 smithi114 ceph-mon[28260]: purged_snaps scrub starts 2024-09-20T19:41:49.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:48 smithi114 ceph-mon[28260]: purged_snaps scrub ok 2024-09-20T19:41:49.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:48 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:41:49.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:48 smithi114 ceph-mon[28260]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:49.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:48 smithi114 ceph-mon[28260]: osd.5 [v2:172.21.15.160:6816/3716955701,v1:172.21.15.160:6817/3716955701] boot 2024-09-20T19:41:49.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:48 smithi114 ceph-mon[28260]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T19:41:49.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:48 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:41:49.265 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:41:49.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:48 smithi160 ceph-mon[35859]: purged_snaps scrub starts 2024-09-20T19:41:49.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:48 smithi160 ceph-mon[35859]: purged_snaps scrub ok 2024-09-20T19:41:49.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:48 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:41:49.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:48 smithi160 ceph-mon[35859]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-20T19:41:49.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:48 smithi160 ceph-mon[35859]: osd.5 [v2:172.21.15.160:6816/3716955701,v1:172.21.15.160:6817/3716955701] boot 2024-09-20T19:41:49.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:48 smithi160 ceph-mon[35859]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T19:41:49.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:48 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T19:41:49.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.710+0000 7f2f4bb88640 1 -- 172.21.15.114:0/3317780167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f44100840 msgr2=0x7f2f44100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:49.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.710+0000 7f2f4bb88640 1 --2- 172.21.15.114:0/3317780167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f44100840 0x7f2f44100c40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009880 tx=0x7f2f3c02a4a0 comp rx=0 tx=0).stop 2024-09-20T19:41:49.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.710+0000 7f2f4bb88640 1 -- 172.21.15.114:0/3317780167 shutdown_connections 2024-09-20T19:41:49.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.710+0000 7f2f4bb88640 1 --2- 172.21.15.114:0/3317780167 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2f44101a40 0x7f2f44101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:49.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.710+0000 7f2f4bb88640 1 --2- 172.21.15.114:0/3317780167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f44100840 0x7f2f44100c40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:49.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.710+0000 7f2f4bb88640 1 -- 172.21.15.114:0/3317780167 >> 172.21.15.114:0/3317780167 conn(0x7f2f440fbff0 msgr2=0x7f2f440fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:49.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.711+0000 7f2f4bb88640 1 -- 172.21.15.114:0/3317780167 shutdown_connections 2024-09-20T19:41:49.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.711+0000 7f2f4bb88640 1 -- 172.21.15.114:0/3317780167 wait complete. 2024-09-20T19:41:49.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.711+0000 7f2f4bb88640 1 Processor -- start 2024-09-20T19:41:49.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.711+0000 7f2f4bb88640 1 -- start start 2024-09-20T19:41:49.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.711+0000 7f2f4bb88640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2f44100840 0x7f2f44197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:49.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.711+0000 7f2f4bb88640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f44101a40 0x7f2f44197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:49.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.711+0000 7f2f4bb88640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2f441096e0 con 0x7f2f44101a40 2024-09-20T19:41:49.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.711+0000 7f2f4bb88640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2f44109560 con 0x7f2f44100840 2024-09-20T19:41:49.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.712+0000 7f2f49123640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f44101a40 0x7f2f44197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:49.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.712+0000 7f2f49123640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f44101a40 0x7f2f44197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45212/0 (socket says 172.21.15.114:45212) 2024-09-20T19:41:49.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.712+0000 7f2f49123640 1 -- 172.21.15.114:0/1603456470 learned_addr learned my addr 172.21.15.114:0/1603456470 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:41:49.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.712+0000 7f2f49924640 1 --2- 172.21.15.114:0/1603456470 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2f44100840 0x7f2f44197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:49.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.712+0000 7f2f49123640 1 -- 172.21.15.114:0/1603456470 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2f44100840 msgr2=0x7f2f44197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:49.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.712+0000 7f2f49123640 1 --2- 172.21.15.114:0/1603456470 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2f44100840 0x7f2f44197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:49.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.712+0000 7f2f49123640 1 -- 172.21.15.114:0/1603456470 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f44198120 con 0x7f2f44101a40 2024-09-20T19:41:49.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.712+0000 7f2f49123640 1 --2- 172.21.15.114:0/1603456470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f44101a40 0x7f2f44197aa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800b420 tx=0x7f2f3800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:49.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.712+0000 7f2f42ffd640 1 -- 172.21.15.114:0/1603456470 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2f38011630 con 0x7f2f44101a40 2024-09-20T19:41:49.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.712+0000 7f2f42ffd640 1 -- 172.21.15.114:0/1603456470 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2f38016070 con 0x7f2f44101a40 2024-09-20T19:41:49.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.712+0000 7f2f4bb88640 1 -- 172.21.15.114:0/1603456470 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f4419cb60 con 0x7f2f44101a40 2024-09-20T19:41:49.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.712+0000 7f2f42ffd640 1 -- 172.21.15.114:0/1603456470 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2f38010d40 con 0x7f2f44101a40 2024-09-20T19:41:49.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.712+0000 7f2f4bb88640 1 -- 172.21.15.114:0/1603456470 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f4419d170 con 0x7f2f44101a40 2024-09-20T19:41:49.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.713+0000 7f2f4bb88640 1 -- 172.21.15.114:0/1603456470 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f44100c40 con 0x7f2f44101a40 2024-09-20T19:41:49.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.714+0000 7f2f42ffd640 1 -- 172.21.15.114:0/1603456470 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f2f3801b460 con 0x7f2f44101a40 2024-09-20T19:41:49.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.714+0000 7f2f42ffd640 1 --2- 172.21.15.114:0/1603456470 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2f2c06e2b0 0x7f2f2c070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:49.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.714+0000 7f2f42ffd640 1 -- 172.21.15.114:0/1603456470 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f2f3808d740 con 0x7f2f44101a40 2024-09-20T19:41:49.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.714+0000 7f2f49924640 1 --2- 172.21.15.114:0/1603456470 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2f2c06e2b0 0x7f2f2c070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:49.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.715+0000 7f2f49924640 1 --2- 172.21.15.114:0/1603456470 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2f2c06e2b0 0x7f2f2c070770 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c000c00 tx=0x7f2f3c005a20 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:49.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.719+0000 7f2f42ffd640 1 -- 172.21.15.114:0/1603456470 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f2f3805b3e0 con 0x7f2f44101a40 2024-09-20T19:41:49.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.895+0000 7f2f4bb88640 1 -- 172.21.15.114:0/1603456470 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f2f4419db00 con 0x7f2f44101a40 2024-09-20T19:41:49.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.895+0000 7f2f42ffd640 1 -- 172.21.15.114:0/1603456470 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v41) ==== 74+0+130 (secure 0 0 0) 0x7f2f3805f090 con 0x7f2f44101a40 2024-09-20T19:41:49.897 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:41:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.897+0000 7f2f4bb88640 1 -- 172.21.15.114:0/1603456470 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2f2c06e2b0 msgr2=0x7f2f2c070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.897+0000 7f2f4bb88640 1 --2- 172.21.15.114:0/1603456470 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2f2c06e2b0 0x7f2f2c070770 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c000c00 tx=0x7f2f3c005a20 comp rx=0 tx=0).stop 2024-09-20T19:41:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.897+0000 7f2f4bb88640 1 -- 172.21.15.114:0/1603456470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f44101a40 msgr2=0x7f2f44197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.897+0000 7f2f4bb88640 1 --2- 172.21.15.114:0/1603456470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f44101a40 0x7f2f44197aa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800b420 tx=0x7f2f3800b8e0 comp rx=0 tx=0).stop 2024-09-20T19:41:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.897+0000 7f2f4bb88640 1 -- 172.21.15.114:0/1603456470 shutdown_connections 2024-09-20T19:41:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.897+0000 7f2f4bb88640 1 --2- 172.21.15.114:0/1603456470 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2f2c06e2b0 0x7f2f2c070770 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.897+0000 7f2f4bb88640 1 --2- 172.21.15.114:0/1603456470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f44101a40 0x7f2f44197aa0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.897+0000 7f2f4bb88640 1 --2- 172.21.15.114:0/1603456470 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2f44100840 0x7f2f44197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.897+0000 7f2f4bb88640 1 -- 172.21.15.114:0/1603456470 >> 172.21.15.114:0/1603456470 conn(0x7f2f440fbff0 msgr2=0x7f2f44069be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.897+0000 7f2f4bb88640 1 -- 172.21.15.114:0/1603456470 shutdown_connections 2024-09-20T19:41:49.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:49.897+0000 7f2f4bb88640 1 -- 172.21.15.114:0/1603456470 wait complete. 2024-09-20T19:41:50.602 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":41,"num_osds":6,"num_up_osds":6,"osd_up_since":1726861308,"num_in_osds":6,"osd_in_since":1726861285,"num_remapped_pgs":0} 2024-09-20T19:41:50.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T19:41:50.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:50 smithi160 ceph-mon[35859]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T19:41:50.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:50 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1603456470' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T19:41:50.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:50 smithi160 ceph-mon[35859]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:41:50.849 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:41:50.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:50 smithi114 ceph-mon[28260]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T19:41:50.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:50 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1603456470' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T19:41:50.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:50 smithi114 ceph-mon[28260]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:41:51.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.290+0000 7ff4a5771640 1 -- 172.21.15.114:0/790866525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4a0100820 msgr2=0x7ff4a0100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:51.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.290+0000 7ff4a5771640 1 --2- 172.21.15.114:0/790866525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4a0100820 0x7ff4a0100c80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff494009880 tx=0x7ff49402a4a0 comp rx=0 tx=0).stop 2024-09-20T19:41:51.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.290+0000 7ff4a5771640 1 -- 172.21.15.114:0/790866525 shutdown_connections 2024-09-20T19:41:51.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.290+0000 7ff4a5771640 1 --2- 172.21.15.114:0/790866525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4a0100820 0x7ff4a0100c80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:51.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.290+0000 7ff4a5771640 1 --2- 172.21.15.114:0/790866525 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff4a0106820 0x7ff4a0106c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:51.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.290+0000 7ff4a5771640 1 -- 172.21.15.114:0/790866525 >> 172.21.15.114:0/790866525 conn(0x7ff4a00fc520 msgr2=0x7ff4a00fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:51.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.290+0000 7ff4a5771640 1 -- 172.21.15.114:0/790866525 shutdown_connections 2024-09-20T19:41:51.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.290+0000 7ff4a5771640 1 -- 172.21.15.114:0/790866525 wait complete. 2024-09-20T19:41:51.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.291+0000 7ff4a5771640 1 Processor -- start 2024-09-20T19:41:51.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.291+0000 7ff4a5771640 1 -- start start 2024-09-20T19:41:51.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.291+0000 7ff4a5771640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff4a0100820 0x7ff4a019d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:51.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.291+0000 7ff4a5771640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4a0106820 0x7ff4a019dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:51.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.291+0000 7ff4a5771640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a010dcf0 con 0x7ff4a0106820 2024-09-20T19:41:51.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.291+0000 7ff4a5771640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a010db70 con 0x7ff4a0100820 2024-09-20T19:41:51.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.291+0000 7ff49e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4a0106820 0x7ff4a019dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:51.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.291+0000 7ff49e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4a0106820 0x7ff4a019dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45232/0 (socket says 172.21.15.114:45232) 2024-09-20T19:41:51.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.291+0000 7ff49e7fc640 1 -- 172.21.15.114:0/4036921543 learned_addr learned my addr 172.21.15.114:0/4036921543 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:41:51.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.291+0000 7ff49effd640 1 --2- 172.21.15.114:0/4036921543 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff4a0100820 0x7ff4a019d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:51.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.291+0000 7ff49e7fc640 1 -- 172.21.15.114:0/4036921543 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff4a0100820 msgr2=0x7ff4a019d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:51.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.291+0000 7ff49e7fc640 1 --2- 172.21.15.114:0/4036921543 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff4a0100820 0x7ff4a019d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:51.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.291+0000 7ff49e7fc640 1 -- 172.21.15.114:0/4036921543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a01977e0 con 0x7ff4a0106820 2024-09-20T19:41:51.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.292+0000 7ff49e7fc640 1 --2- 172.21.15.114:0/4036921543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4a0106820 0x7ff4a019dba0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff494031460 tx=0x7ff4940045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:51.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.292+0000 7ff487fff640 1 -- 172.21.15.114:0/4036921543 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff49403f070 con 0x7ff4a0106820 2024-09-20T19:41:51.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.292+0000 7ff4a5771640 1 -- 172.21.15.114:0/4036921543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a0197a70 con 0x7ff4a0106820 2024-09-20T19:41:51.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.292+0000 7ff487fff640 1 -- 172.21.15.114:0/4036921543 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff4940035f0 con 0x7ff4a0106820 2024-09-20T19:41:51.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.292+0000 7ff487fff640 1 -- 172.21.15.114:0/4036921543 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff494035030 con 0x7ff4a0106820 2024-09-20T19:41:51.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.292+0000 7ff4a5771640 1 -- 172.21.15.114:0/4036921543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a0197fa0 con 0x7ff4a0106820 2024-09-20T19:41:51.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.293+0000 7ff4a5771640 1 -- 172.21.15.114:0/4036921543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4a0101f60 con 0x7ff4a0106820 2024-09-20T19:41:51.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.293+0000 7ff487fff640 1 -- 172.21.15.114:0/4036921543 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7ff494042070 con 0x7ff4a0106820 2024-09-20T19:41:51.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.294+0000 7ff487fff640 1 --2- 172.21.15.114:0/4036921543 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff48806e2b0 0x7ff488070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:51.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.294+0000 7ff487fff640 1 -- 172.21.15.114:0/4036921543 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7ff4940ae740 con 0x7ff4a0106820 2024-09-20T19:41:51.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.294+0000 7ff49effd640 1 --2- 172.21.15.114:0/4036921543 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff48806e2b0 0x7ff488070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:51.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.294+0000 7ff49effd640 1 --2- 172.21.15.114:0/4036921543 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff48806e2b0 0x7ff488070770 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff490005fd0 tx=0x7ff49000b390 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:51.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.298+0000 7ff487fff640 1 -- 172.21.15.114:0/4036921543 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7ff494080090 con 0x7ff4a0106820 2024-09-20T19:41:51.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.466+0000 7ff4a5771640 1 -- 172.21.15.114:0/4036921543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff4a0109da0 con 0x7ff4a0106820 2024-09-20T19:41:51.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.467+0000 7ff487fff640 1 -- 172.21.15.114:0/4036921543 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+11418 (secure 0 0 0) 0x7ff4940848c0 con 0x7ff4a0106820 2024-09-20T19:41:51.469 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:41:51.469 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":41,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","created":"2024-09-20T19:35:57.433837+0000","modified":"2024-09-20T19:41:49.627858+0000","last_up_change":"2024-09-20T19:41:48.626363+0000","last_in_change":"2024-09-20T19:41:25.231165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T19:40:32.545286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9ac4e0d5-030f-4de4-96a7-fd80f6c10a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":928370375},{"type":"v1","addr":"172.21.15.114:6803","nonce":928370375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":928370375},{"type":"v1","addr":"172.21.15.114:6805","nonce":928370375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":928370375},{"type":"v1","addr":"172.21.15.114:6809","nonce":928370375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":928370375},{"type":"v1","addr":"172.21.15.114:6807","nonce":928370375}]},"public_addr":"172.21.15.114:6803/928370375","cluster_addr":"172.21.15.114:6805/928370375","heartbeat_back_addr":"172.21.15.114:6809/928370375","heartbeat_front_addr":"172.21.15.114:6807/928370375","state":["exists","up"]},{"osd":1,"uuid":"98aef96e-fc81-45ad-9cc1-49ce9ba617b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":326057805},{"type":"v1","addr":"172.21.15.114:6811","nonce":326057805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":326057805},{"type":"v1","addr":"172.21.15.114:6813","nonce":326057805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":326057805},{"type":"v1","addr":"172.21.15.114:6817","nonce":326057805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":326057805},{"type":"v1","addr":"172.21.15.114:6815","nonce":326057805}]},"public_addr":"172.21.15.114:6811/326057805","cluster_addr":"172.21.15.114:6813/326057805","heartbeat_back_addr":"172.21.15.114:6817/326057805","heartbeat_front_addr":"172.21.15.114:6815/326057805","state":["exists","up"]},{"osd":2,"uuid":"89c2fdec-88bf-4bcb-94be-fef898abddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4082826493},{"type":"v1","addr":"172.21.15.114:6819","nonce":4082826493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4082826493},{"type":"v1","addr":"172.21.15.114:6821","nonce":4082826493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4082826493},{"type":"v1","addr":"172.21.15.114:6825","nonce":4082826493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4082826493},{"type":"v1","addr":"172.21.15.114:6823","nonce":4082826493}]},"public_addr":"172.21.15.114:6819/4082826493","cluster_addr":"172.21.15.114:6821/4082826493","heartbeat_back_addr":"172.21.15.114:6825/4082826493","heartbeat_front_addr":"172.21.15.114:6823/4082826493","state":["exists","up"]},{"osd":3,"uuid":"64f8a718-5814-4210-8696-b2acd153c046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":496816402},{"type":"v1","addr":"172.21.15.160:6801","nonce":496816402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":496816402},{"type":"v1","addr":"172.21.15.160:6803","nonce":496816402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":496816402},{"type":"v1","addr":"172.21.15.160:6807","nonce":496816402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":496816402},{"type":"v1","addr":"172.21.15.160:6805","nonce":496816402}]},"public_addr":"172.21.15.160:6801/496816402","cluster_addr":"172.21.15.160:6803/496816402","heartbeat_back_addr":"172.21.15.160:6807/496816402","heartbeat_front_addr":"172.21.15.160:6805/496816402","state":["exists","up"]},{"osd":4,"uuid":"302c7425-1a84-44fc-a4c9-58a71a9cdc54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3464010154},{"type":"v1","addr":"172.21.15.160:6809","nonce":3464010154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3464010154},{"type":"v1","addr":"172.21.15.160:6811","nonce":3464010154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3464010154},{"type":"v1","addr":"172.21.15.160:6815","nonce":3464010154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3464010154},{"type":"v1","addr":"172.21.15.160:6813","nonce":3464010154}]},"public_addr":"172.21.15.160:6809/3464010154","cluster_addr":"172.21.15.160:6811/3464010154","heartbeat_back_addr":"172.21.15.160:6815/3464010154","heartbeat_front_addr":"172.21.15.160:6813/3464010154","state":["exists","up"]},{"osd":5,"uuid":"a645a156-dc73-4033-adde-4d2380a9c000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":3716955701},{"type":"v1","addr":"172.21.15.160:6817","nonce":3716955701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":3716955701},{"type":"v1","addr":"172.21.15.160:6819","nonce":3716955701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":3716955701},{"type":"v1","addr":"172.21.15.160:6823","nonce":3716955701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":3716955701},{"type":"v1","addr":"172.21.15.160:6821","nonce":3716955701}]},"public_addr":"172.21.15.160:6817/3716955701","cluster_addr":"172.21.15.160:6819/3716955701","heartbeat_back_addr":"172.21.15.160:6823/3716955701","heartbeat_front_addr":"172.21.15.160:6821/3716955701","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:39:31.890087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:39:59.898001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:40:28.463295+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:40:53.291600+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:41:20.621298+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:41:46.675541+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/1191801334":"2024-09-21T19:37:56.220717+0000","172.21.15.114:0/3994747756":"2024-09-21T19:37:56.220717+0000","172.21.15.114:0/3541238036":"2024-09-21T19:36:27.357221+0000","172.21.15.114:0/62878745":"2024-09-21T19:36:27.357221+0000","172.21.15.114:0/4279612706":"2024-09-21T19:36:27.357221+0000","172.21.15.114:0/2669286306":"2024-09-21T19:36:58.132711+0000","172.21.15.114:6800/689537755":"2024-09-21T19:36:27.357221+0000","172.21.15.114:6801/689537755":"2024-09-21T19:36:27.357221+0000","172.21.15.114:6801/2644929394":"2024-09-21T19:36:58.132711+0000","172.21.15.114:0/1230605596":"2024-09-21T19:36:58.132711+0000","172.21.15.114:6800/969843775":"2024-09-21T19:37:56.220717+0000","172.21.15.114:6800/2644929394":"2024-09-21T19:36:58.132711+0000","172.21.15.114:0/2896121214":"2024-09-21T19:36:58.132711+0000","172.21.15.114:6801/969843775":"2024-09-21T19:37:56.220717+0000","172.21.15.114:0/2774428689":"2024-09-21T19:37:56.220717+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T19:41:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.469+0000 7ff4a5771640 1 -- 172.21.15.114:0/4036921543 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff48806e2b0 msgr2=0x7ff488070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.469+0000 7ff4a5771640 1 --2- 172.21.15.114:0/4036921543 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff48806e2b0 0x7ff488070770 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff490005fd0 tx=0x7ff49000b390 comp rx=0 tx=0).stop 2024-09-20T19:41:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.469+0000 7ff4a5771640 1 -- 172.21.15.114:0/4036921543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4a0106820 msgr2=0x7ff4a019dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.469+0000 7ff4a5771640 1 --2- 172.21.15.114:0/4036921543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4a0106820 0x7ff4a019dba0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff494031460 tx=0x7ff4940045e0 comp rx=0 tx=0).stop 2024-09-20T19:41:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.469+0000 7ff4a5771640 1 -- 172.21.15.114:0/4036921543 shutdown_connections 2024-09-20T19:41:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.469+0000 7ff4a5771640 1 --2- 172.21.15.114:0/4036921543 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff48806e2b0 0x7ff488070770 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.469+0000 7ff4a5771640 1 --2- 172.21.15.114:0/4036921543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4a0106820 0x7ff4a019dba0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.469+0000 7ff4a5771640 1 --2- 172.21.15.114:0/4036921543 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff4a0100820 0x7ff4a019d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.469+0000 7ff4a5771640 1 -- 172.21.15.114:0/4036921543 >> 172.21.15.114:0/4036921543 conn(0x7ff4a00fc520 msgr2=0x7ff4a006a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.469+0000 7ff4a5771640 1 -- 172.21.15.114:0/4036921543 shutdown_connections 2024-09-20T19:41:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:51.469+0000 7ff4a5771640 1 -- 172.21.15.114:0/4036921543 wait complete. 2024-09-20T19:41:51.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:51 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/4036921543' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T19:41:52.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:51 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/4036921543' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T19:41:52.179 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T19:40:32.545286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-20T19:41:52.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-20T19:41:52.427 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:41:52.819 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:52 smithi114 ceph-mon[28260]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:41:52.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.901+0000 7f4207a43640 1 -- 172.21.15.114:0/4088819365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42001067e0 msgr2=0x7f4200106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:52.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.901+0000 7f4207a43640 1 --2- 172.21.15.114:0/4088819365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42001067e0 0x7f4200106bc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f41f0009c20 tx=0x7f41f002a970 comp rx=0 tx=0).stop 2024-09-20T19:41:52.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.901+0000 7f4207a43640 1 -- 172.21.15.114:0/4088819365 shutdown_connections 2024-09-20T19:41:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.901+0000 7f4207a43640 1 --2- 172.21.15.114:0/4088819365 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f42001007e0 0x7f4200100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.901+0000 7f4207a43640 1 --2- 172.21.15.114:0/4088819365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42001067e0 0x7f4200106bc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.901+0000 7f4207a43640 1 -- 172.21.15.114:0/4088819365 >> 172.21.15.114:0/4088819365 conn(0x7f42000fc4c0 msgr2=0x7f42000fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.901+0000 7f4207a43640 1 -- 172.21.15.114:0/4088819365 shutdown_connections 2024-09-20T19:41:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.901+0000 7f4207a43640 1 -- 172.21.15.114:0/4088819365 wait complete. 2024-09-20T19:41:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.901+0000 7f4207a43640 1 Processor -- start 2024-09-20T19:41:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.901+0000 7f4207a43640 1 -- start start 2024-09-20T19:41:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f4207a43640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42001007e0 0x7f420019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f4207a43640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f42001067e0 0x7f420019cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f4207a43640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f420010dd10 con 0x7f42001007e0 2024-09-20T19:41:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f4207a43640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f420010db90 con 0x7f42001067e0 2024-09-20T19:41:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f42057df640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42001007e0 0x7f420019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f42057df640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42001007e0 0x7f420019c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45242/0 (socket says 172.21.15.114:45242) 2024-09-20T19:41:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f42057df640 1 -- 172.21.15.114:0/331752374 learned_addr learned my addr 172.21.15.114:0/331752374 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:41:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f4204fde640 1 --2- 172.21.15.114:0/331752374 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f42001067e0 0x7f420019cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f42057df640 1 -- 172.21.15.114:0/331752374 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f42001067e0 msgr2=0x7f420019cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f42057df640 1 --2- 172.21.15.114:0/331752374 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f42001067e0 0x7f420019cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f42057df640 1 -- 172.21.15.114:0/331752374 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42001967e0 con 0x7f42001007e0 2024-09-20T19:41:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f42057df640 1 --2- 172.21.15.114:0/331752374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42001007e0 0x7f420019c660 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f41f0005cc0 tx=0x7f41f002f5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f41fe7fc640 1 -- 172.21.15.114:0/331752374 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f41f0047070 con 0x7f42001007e0 2024-09-20T19:41:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f41fe7fc640 1 -- 172.21.15.114:0/331752374 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f41f002f7e0 con 0x7f42001007e0 2024-09-20T19:41:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f4207a43640 1 -- 172.21.15.114:0/331752374 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4200196a70 con 0x7f42001007e0 2024-09-20T19:41:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.902+0000 7f41fe7fc640 1 -- 172.21.15.114:0/331752374 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f41f0034070 con 0x7f42001007e0 2024-09-20T19:41:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.903+0000 7f4207a43640 1 -- 172.21.15.114:0/331752374 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4200196fa0 con 0x7f42001007e0 2024-09-20T19:41:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.903+0000 7f4207a43640 1 -- 172.21.15.114:0/331752374 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4200101f20 con 0x7f42001007e0 2024-09-20T19:41:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.904+0000 7f41fe7fc640 1 -- 172.21.15.114:0/331752374 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f41f004a070 con 0x7f42001007e0 2024-09-20T19:41:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.904+0000 7f41fe7fc640 1 --2- 172.21.15.114:0/331752374 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f41e806e2b0 0x7f41e8070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.904+0000 7f41fe7fc640 1 -- 172.21.15.114:0/331752374 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f41f00aee20 con 0x7f42001007e0 2024-09-20T19:41:52.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.904+0000 7f4204fde640 1 --2- 172.21.15.114:0/331752374 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f41e806e2b0 0x7f41e8070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:52.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.905+0000 7f4204fde640 1 --2- 172.21.15.114:0/331752374 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f41e806e2b0 0x7f41e8070770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4200105e90 tx=0x7f41f4009110 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:52.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:52.909+0000 7f41fe7fc640 1 -- 172.21.15.114:0/331752374 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f41f003f070 con 0x7f42001007e0 2024-09-20T19:41:53.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:52 smithi160 ceph-mon[35859]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:41:53.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:53.075+0000 7f4207a43640 1 -- 172.21.15.114:0/331752374 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4200197d10 con 0x7f42001007e0 2024-09-20T19:41:53.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:53.076+0000 7f41fe7fc640 1 -- 172.21.15.114:0/331752374 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7f41f007cbe0 con 0x7f42001007e0 2024-09-20T19:41:53.078 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-09-20T19:41:53.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:53.078+0000 7f4207a43640 1 -- 172.21.15.114:0/331752374 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f41e806e2b0 msgr2=0x7f41e8070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:53.078+0000 7f4207a43640 1 --2- 172.21.15.114:0/331752374 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f41e806e2b0 0x7f41e8070770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4200105e90 tx=0x7f41f4009110 comp rx=0 tx=0).stop 2024-09-20T19:41:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:53.078+0000 7f4207a43640 1 -- 172.21.15.114:0/331752374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42001007e0 msgr2=0x7f420019c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:53.078+0000 7f4207a43640 1 --2- 172.21.15.114:0/331752374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42001007e0 0x7f420019c660 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f41f0005cc0 tx=0x7f41f002f5f0 comp rx=0 tx=0).stop 2024-09-20T19:41:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:53.078+0000 7f4207a43640 1 -- 172.21.15.114:0/331752374 shutdown_connections 2024-09-20T19:41:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:53.078+0000 7f4207a43640 1 --2- 172.21.15.114:0/331752374 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f41e806e2b0 0x7f41e8070770 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:53.078+0000 7f4207a43640 1 --2- 172.21.15.114:0/331752374 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f42001067e0 0x7f420019cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:53.078+0000 7f4207a43640 1 --2- 172.21.15.114:0/331752374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42001007e0 0x7f420019c660 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:53.078+0000 7f4207a43640 1 -- 172.21.15.114:0/331752374 >> 172.21.15.114:0/331752374 conn(0x7f42000fc4c0 msgr2=0x7f42000fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:53.078+0000 7f4207a43640 1 -- 172.21.15.114:0/331752374 shutdown_connections 2024-09-20T19:41:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:53.078+0000 7f4207a43640 1 -- 172.21.15.114:0/331752374 wait complete. 2024-09-20T19:41:53.767 INFO:tasks.cephadm:Setting up client nodes... 2024-09-20T19:41:53.767 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-20T19:41:53.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:53 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/331752374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T19:41:54.012 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:41:54.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:53 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/331752374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T19:41:54.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.457+0000 7f0f2e6af640 1 -- 172.21.15.114:0/3935822948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f28100880 msgr2=0x7f0f28100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.457+0000 7f0f2e6af640 1 --2- 172.21.15.114:0/3935822948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f28100880 0x7f0f28100c80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009880 tx=0x7f0f1802a4a0 comp rx=0 tx=0).stop 2024-09-20T19:41:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.457+0000 7f0f2e6af640 1 -- 172.21.15.114:0/3935822948 shutdown_connections 2024-09-20T19:41:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.457+0000 7f0f2e6af640 1 --2- 172.21.15.114:0/3935822948 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0f28101a80 0x7f0f28101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.457+0000 7f0f2e6af640 1 --2- 172.21.15.114:0/3935822948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f28100880 0x7f0f28100c80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.457+0000 7f0f2e6af640 1 -- 172.21.15.114:0/3935822948 >> 172.21.15.114:0/3935822948 conn(0x7f0f280fc010 msgr2=0x7f0f280fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.458+0000 7f0f2e6af640 1 -- 172.21.15.114:0/3935822948 shutdown_connections 2024-09-20T19:41:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.458+0000 7f0f2e6af640 1 -- 172.21.15.114:0/3935822948 wait complete. 2024-09-20T19:41:54.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.458+0000 7f0f2e6af640 1 Processor -- start 2024-09-20T19:41:54.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.458+0000 7f0f2e6af640 1 -- start start 2024-09-20T19:41:54.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.458+0000 7f0f2e6af640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0f28100880 0x7f0f281974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:54.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.458+0000 7f0f2e6af640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f28101a80 0x7f0f28197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:54.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.458+0000 7f0f2e6af640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0f281096a0 con 0x7f0f28101a80 2024-09-20T19:41:54.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.458+0000 7f0f2e6af640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0f28109520 con 0x7f0f28100880 2024-09-20T19:41:54.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.459+0000 7f0f277fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f28101a80 0x7f0f28197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:54.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.459+0000 7f0f277fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f28101a80 0x7f0f28197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45264/0 (socket says 172.21.15.114:45264) 2024-09-20T19:41:54.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.459+0000 7f0f277fe640 1 -- 172.21.15.114:0/3562421199 learned_addr learned my addr 172.21.15.114:0/3562421199 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:41:54.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.459+0000 7f0f27fff640 1 --2- 172.21.15.114:0/3562421199 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0f28100880 0x7f0f281974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:54.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.459+0000 7f0f277fe640 1 -- 172.21.15.114:0/3562421199 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0f28100880 msgr2=0x7f0f281974f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:54.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.459+0000 7f0f277fe640 1 --2- 172.21.15.114:0/3562421199 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0f28100880 0x7f0f281974f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:54.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.459+0000 7f0f277fe640 1 -- 172.21.15.114:0/3562421199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f281980b0 con 0x7f0f28101a80 2024-09-20T19:41:54.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.459+0000 7f0f277fe640 1 --2- 172.21.15.114:0/3562421199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f28101a80 0x7f0f28197a30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c00d8d0 tx=0x7f0f1c00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:54.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.459+0000 7f0f257fa640 1 -- 172.21.15.114:0/3562421199 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0f1c017070 con 0x7f0f28101a80 2024-09-20T19:41:54.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.459+0000 7f0f257fa640 1 -- 172.21.15.114:0/3562421199 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0f1c010460 con 0x7f0f28101a80 2024-09-20T19:41:54.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.459+0000 7f0f2e6af640 1 -- 172.21.15.114:0/3562421199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f2819caf0 con 0x7f0f28101a80 2024-09-20T19:41:54.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.459+0000 7f0f257fa640 1 -- 172.21.15.114:0/3562421199 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0f1c00f5f0 con 0x7f0f28101a80 2024-09-20T19:41:54.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.459+0000 7f0f2e6af640 1 -- 172.21.15.114:0/3562421199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f2819d080 con 0x7f0f28101a80 2024-09-20T19:41:54.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.460+0000 7f0f2e6af640 1 -- 172.21.15.114:0/3562421199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f28100c80 con 0x7f0f28101a80 2024-09-20T19:41:54.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.461+0000 7f0f257fa640 1 -- 172.21.15.114:0/3562421199 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f0f1c00f8b0 con 0x7f0f28101a80 2024-09-20T19:41:54.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.461+0000 7f0f257fa640 1 --2- 172.21.15.114:0/3562421199 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0f1006e2b0 0x7f0f10070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:54.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.461+0000 7f0f27fff640 1 --2- 172.21.15.114:0/3562421199 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0f1006e2b0 0x7f0f10070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:54.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.461+0000 7f0f257fa640 1 -- 172.21.15.114:0/3562421199 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f0f1c08dc50 con 0x7f0f28101a80 2024-09-20T19:41:54.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.462+0000 7f0f27fff640 1 --2- 172.21.15.114:0/3562421199 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0f1006e2b0 0x7f0f10070770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009230 tx=0x7f0f180058d0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:54.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.466+0000 7f0f257fa640 1 -- 172.21.15.114:0/3562421199 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f0f1c05f5a0 con 0x7f0f28101a80 2024-09-20T19:41:54.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.701+0000 7f0f2e6af640 1 -- 172.21.15.114:0/3562421199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f0f2819d610 con 0x7f0f28101a80 2024-09-20T19:41:54.703 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:54 smithi114 ceph-mon[28260]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:41:54.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.703+0000 7f0f257fa640 1 -- 172.21.15.114:0/3562421199 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) ==== 170+0+59 (secure 0 0 0) 0x7f0f1c063dd0 con 0x7f0f28101a80 2024-09-20T19:41:54.705 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-09-20T19:41:54.705 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAC0O1mnCnlKRAA+8kCEYn/5aFDqecuPgvb/Q== 2024-09-20T19:41:54.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.706+0000 7f0f2e6af640 1 -- 172.21.15.114:0/3562421199 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0f1006e2b0 msgr2=0x7f0f10070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:54.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.706+0000 7f0f2e6af640 1 --2- 172.21.15.114:0/3562421199 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0f1006e2b0 0x7f0f10070770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009230 tx=0x7f0f180058d0 comp rx=0 tx=0).stop 2024-09-20T19:41:54.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.706+0000 7f0f2e6af640 1 -- 172.21.15.114:0/3562421199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f28101a80 msgr2=0x7f0f28197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:54.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.706+0000 7f0f2e6af640 1 --2- 172.21.15.114:0/3562421199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f28101a80 0x7f0f28197a30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c00d8d0 tx=0x7f0f1c00dd90 comp rx=0 tx=0).stop 2024-09-20T19:41:54.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.706+0000 7f0f2e6af640 1 -- 172.21.15.114:0/3562421199 shutdown_connections 2024-09-20T19:41:54.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.706+0000 7f0f2e6af640 1 --2- 172.21.15.114:0/3562421199 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0f1006e2b0 0x7f0f10070770 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:54.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.706+0000 7f0f2e6af640 1 --2- 172.21.15.114:0/3562421199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f28101a80 0x7f0f28197a30 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:54.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.706+0000 7f0f2e6af640 1 --2- 172.21.15.114:0/3562421199 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0f28100880 0x7f0f281974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:54.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.706+0000 7f0f2e6af640 1 -- 172.21.15.114:0/3562421199 >> 172.21.15.114:0/3562421199 conn(0x7f0f280fc010 msgr2=0x7f0f28069ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:54.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.706+0000 7f0f2e6af640 1 -- 172.21.15.114:0/3562421199 shutdown_connections 2024-09-20T19:41:54.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:54.706+0000 7f0f2e6af640 1 -- 172.21.15.114:0/3562421199 wait complete. 2024-09-20T19:41:55.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:54 smithi160 ceph-mon[35859]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:41:55.368 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:41:55.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-20T19:41:55.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-20T19:41:55.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-20T19:41:55.665 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi160/config 2024-09-20T19:41:55.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:55 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/3562421199' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:41:55.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:55 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/3562421199' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:41:55.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:55 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3562421199' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:41:55.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:55 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3562421199' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:41:56.138 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.127+0000 7f1c369ac640 1 -- 172.21.15.160:0/363548838 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c30101980 msgr2=0x7f1c30101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:56.138 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.127+0000 7f1c369ac640 1 --2- 172.21.15.160:0/363548838 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c30101980 0x7f1c30101d80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009880 tx=0x7f1c2002a4a0 comp rx=0 tx=0).stop 2024-09-20T19:41:56.138 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.127+0000 7f1c369ac640 1 -- 172.21.15.160:0/363548838 shutdown_connections 2024-09-20T19:41:56.138 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.127+0000 7f1c369ac640 1 --2- 172.21.15.160:0/363548838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c30102b80 0x7f1c30103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:56.138 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.127+0000 7f1c369ac640 1 --2- 172.21.15.160:0/363548838 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c30101980 0x7f1c30101d80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:56.139 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.127+0000 7f1c369ac640 1 -- 172.21.15.160:0/363548838 >> 172.21.15.160:0/363548838 conn(0x7f1c300fd110 msgr2=0x7f1c300ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:56.139 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.127+0000 7f1c369ac640 1 -- 172.21.15.160:0/363548838 shutdown_connections 2024-09-20T19:41:56.139 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.127+0000 7f1c369ac640 1 -- 172.21.15.160:0/363548838 wait complete. 2024-09-20T19:41:56.139 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.127+0000 7f1c369ac640 1 Processor -- start 2024-09-20T19:41:56.139 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.127+0000 7f1c369ac640 1 -- start start 2024-09-20T19:41:56.139 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.127+0000 7f1c369ac640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c30101980 0x7f1c30197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:56.139 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.127+0000 7f1c369ac640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c30102b80 0x7f1c30197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:56.139 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.127+0000 7f1c369ac640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3010a820 con 0x7f1c30102b80 2024-09-20T19:41:56.139 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.127+0000 7f1c369ac640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3010a6a0 con 0x7f1c30101980 2024-09-20T19:41:56.139 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.127+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c30101980 0x7f1c30197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:56.139 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.128+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c30101980 0x7f1c30197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:48758/0 (socket says 172.21.15.160:48758) 2024-09-20T19:41:56.139 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.128+0000 7f1c2ffff640 1 -- 172.21.15.160:0/2190712010 learned_addr learned my addr 172.21.15.160:0/2190712010 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-09-20T19:41:56.139 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.128+0000 7f1c2ffff640 1 -- 172.21.15.160:0/2190712010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c30102b80 msgr2=0x7f1c30197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:56.140 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.128+0000 7f1c2ffff640 1 --2- 172.21.15.160:0/2190712010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c30102b80 0x7f1c30197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:56.140 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.128+0000 7f1c2ffff640 1 -- 172.21.15.160:0/2190712010 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c301980f0 con 0x7f1c30101980 2024-09-20T19:41:56.140 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.128+0000 7f1c2ffff640 1 --2- 172.21.15.160:0/2190712010 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c30101980 0x7f1c30197530 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1c20006010 tx=0x7f1c200048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:56.140 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.128+0000 7f1c2dffb640 1 -- 172.21.15.160:0/2190712010 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f1c2003f070 con 0x7f1c30101980 2024-09-20T19:41:56.140 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.128+0000 7f1c2dffb640 1 -- 172.21.15.160:0/2190712010 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1c200044d0 con 0x7f1c30101980 2024-09-20T19:41:56.140 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.128+0000 7f1c369ac640 1 -- 172.21.15.160:0/2190712010 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c3019cad0 con 0x7f1c30101980 2024-09-20T19:41:56.140 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.128+0000 7f1c2dffb640 1 -- 172.21.15.160:0/2190712010 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f1c20035030 con 0x7f1c30101980 2024-09-20T19:41:56.140 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.128+0000 7f1c369ac640 1 -- 172.21.15.160:0/2190712010 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c3019d0e0 con 0x7f1c30101980 2024-09-20T19:41:56.140 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.129+0000 7f1c369ac640 1 -- 172.21.15.160:0/2190712010 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c30101d80 con 0x7f1c30101980 2024-09-20T19:41:56.140 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.130+0000 7f1c2dffb640 1 -- 172.21.15.160:0/2190712010 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f1c20042070 con 0x7f1c30101980 2024-09-20T19:41:56.140 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.130+0000 7f1c2dffb640 1 --2- 172.21.15.160:0/2190712010 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f1c1c06e310 0x7f1c1c0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:56.140 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.130+0000 7f1c2dffb640 1 -- 172.21.15.160:0/2190712010 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f1c200ae9e0 con 0x7f1c30101980 2024-09-20T19:41:56.140 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.130+0000 7f1c27fff640 1 --2- 172.21.15.160:0/2190712010 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f1c1c06e310 0x7f1c1c0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:56.140 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.131+0000 7f1c27fff640 1 --2- 172.21.15.160:0/2190712010 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f1c1c06e310 0x7f1c1c0707d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1c301989e0 tx=0x7f1c18006c90 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:56.141 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.135+0000 7f1c2dffb640 1 -- 172.21.15.160:0/2190712010 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f1c2007c700 con 0x7f1c30101980 2024-09-20T19:41:56.364 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.363+0000 7f1c369ac640 1 -- 172.21.15.160:0/2190712010 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f1c30191630 con 0x7f1c30101980 2024-09-20T19:41:56.369 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.367+0000 7f1c2dffb640 1 -- 172.21.15.160:0/2190712010 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) ==== 170+0+59 (secure 0 0 0) 0x7f1c200803b0 con 0x7f1c30101980 2024-09-20T19:41:56.369 INFO:teuthology.orchestra.run.smithi160.stdout:[client.1] 2024-09-20T19:41:56.369 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQAE0O1m9drBFRAAFZVhMkiOVtbsRiGRF35V0g== 2024-09-20T19:41:56.371 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.369+0000 7f1c369ac640 1 -- 172.21.15.160:0/2190712010 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f1c1c06e310 msgr2=0x7f1c1c0707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:56.371 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.369+0000 7f1c369ac640 1 --2- 172.21.15.160:0/2190712010 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f1c1c06e310 0x7f1c1c0707d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1c301989e0 tx=0x7f1c18006c90 comp rx=0 tx=0).stop 2024-09-20T19:41:56.371 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.369+0000 7f1c369ac640 1 -- 172.21.15.160:0/2190712010 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c30101980 msgr2=0x7f1c30197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:56.371 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.369+0000 7f1c369ac640 1 --2- 172.21.15.160:0/2190712010 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c30101980 0x7f1c30197530 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1c20006010 tx=0x7f1c200048f0 comp rx=0 tx=0).stop 2024-09-20T19:41:56.371 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.370+0000 7f1c369ac640 1 -- 172.21.15.160:0/2190712010 shutdown_connections 2024-09-20T19:41:56.372 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.370+0000 7f1c369ac640 1 --2- 172.21.15.160:0/2190712010 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f1c1c06e310 0x7f1c1c0707d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:56.372 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.370+0000 7f1c369ac640 1 --2- 172.21.15.160:0/2190712010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c30102b80 0x7f1c30197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:56.372 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.370+0000 7f1c369ac640 1 --2- 172.21.15.160:0/2190712010 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c30101980 0x7f1c30197530 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:56.372 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.370+0000 7f1c369ac640 1 -- 172.21.15.160:0/2190712010 >> 172.21.15.160:0/2190712010 conn(0x7f1c300fd110 msgr2=0x7f1c30106090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:56.372 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.370+0000 7f1c369ac640 1 -- 172.21.15.160:0/2190712010 shutdown_connections 2024-09-20T19:41:56.372 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:41:56.370+0000 7f1c369ac640 1 -- 172.21.15.160:0/2190712010 wait complete. 2024-09-20T19:41:56.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:56 smithi114 ceph-mon[28260]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:41:56.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:56 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:41:56.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:56 smithi114 ceph-mon[28260]: from='client.? 172.21.15.160:0/2190712010' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:41:56.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:56 smithi114 ceph-mon[28260]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:41:56.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:56 smithi114 ceph-mon[28260]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:41:57.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:56 smithi160 ceph-mon[35859]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:41:57.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:56 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:41:57.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:56 smithi160 ceph-mon[35859]: from='client.? 172.21.15.160:0/2190712010' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:41:57.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:56 smithi160 ceph-mon[35859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T19:41:57.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:56 smithi160 ceph-mon[35859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T19:41:57.097 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-20T19:41:57.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-20T19:41:57.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-20T19:41:57.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph config log 1 --format=json 2024-09-20T19:41:57.412 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:41:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.873+0000 7f7cbc72e640 1 -- 172.21.15.114:0/3959848985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb4106800 msgr2=0x7f7cb4106be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.873+0000 7f7cbc72e640 1 --2- 172.21.15.114:0/3959848985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb4106800 0x7f7cb4106be0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009880 tx=0x7f7cb002a4a0 comp rx=0 tx=0).stop 2024-09-20T19:41:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.873+0000 7f7cbc72e640 1 -- 172.21.15.114:0/3959848985 shutdown_connections 2024-09-20T19:41:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.873+0000 7f7cbc72e640 1 --2- 172.21.15.114:0/3959848985 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cb4100800 0x7f7cb4100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.873+0000 7f7cbc72e640 1 --2- 172.21.15.114:0/3959848985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb4106800 0x7f7cb4106be0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.873+0000 7f7cbc72e640 1 -- 172.21.15.114:0/3959848985 >> 172.21.15.114:0/3959848985 conn(0x7f7cb40fc4e0 msgr2=0x7f7cb40fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.873+0000 7f7cbc72e640 1 -- 172.21.15.114:0/3959848985 shutdown_connections 2024-09-20T19:41:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.873+0000 7f7cbc72e640 1 -- 172.21.15.114:0/3959848985 wait complete. 2024-09-20T19:41:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.873+0000 7f7cbc72e640 1 Processor -- start 2024-09-20T19:41:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.874+0000 7f7cbc72e640 1 -- start start 2024-09-20T19:41:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.874+0000 7f7cbc72e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cb4100800 0x7f7cb419d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.874+0000 7f7cbc72e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb4106800 0x7f7cb419db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.874+0000 7f7cbc72e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb410dcd0 con 0x7f7cb4106800 2024-09-20T19:41:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.874+0000 7f7cbc72e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb410db50 con 0x7f7cb4100800 2024-09-20T19:41:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.874+0000 7f7cb9cc9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb4106800 0x7f7cb419db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.874+0000 7f7cb9cc9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb4106800 0x7f7cb419db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52840/0 (socket says 172.21.15.114:52840) 2024-09-20T19:41:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.874+0000 7f7cb9cc9640 1 -- 172.21.15.114:0/1810203639 learned_addr learned my addr 172.21.15.114:0/1810203639 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:41:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.874+0000 7f7cba4ca640 1 --2- 172.21.15.114:0/1810203639 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cb4100800 0x7f7cb419d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.874+0000 7f7cb9cc9640 1 -- 172.21.15.114:0/1810203639 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cb4100800 msgr2=0x7f7cb419d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.874+0000 7f7cb9cc9640 1 --2- 172.21.15.114:0/1810203639 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cb4100800 0x7f7cb419d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.874+0000 7f7cb9cc9640 1 -- 172.21.15.114:0/1810203639 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb41977a0 con 0x7f7cb4106800 2024-09-20T19:41:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.874+0000 7f7cba4ca640 1 --2- 172.21.15.114:0/1810203639 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cb4100800 0x7f7cb419d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-20T19:41:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.874+0000 7f7cb9cc9640 1 --2- 172.21.15.114:0/1810203639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb4106800 0x7f7cb419db60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800d6f0 tx=0x7f7ca800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.874+0000 7f7ca77fe640 1 -- 172.21.15.114:0/1810203639 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f7ca8017070 con 0x7f7cb4106800 2024-09-20T19:41:57.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.875+0000 7f7cbc72e640 1 -- 172.21.15.114:0/1810203639 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb4197a90 con 0x7f7cb4106800 2024-09-20T19:41:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.875+0000 7f7ca77fe640 1 -- 172.21.15.114:0/1810203639 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7ca800de00 con 0x7f7cb4106800 2024-09-20T19:41:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.875+0000 7f7ca77fe640 1 -- 172.21.15.114:0/1810203639 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f7ca8004bb0 con 0x7f7cb4106800 2024-09-20T19:41:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.875+0000 7f7cbc72e640 1 -- 172.21.15.114:0/1810203639 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb4198020 con 0x7f7cb4106800 2024-09-20T19:41:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.876+0000 7f7cbc72e640 1 -- 172.21.15.114:0/1810203639 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cb4101f40 con 0x7f7cb4106800 2024-09-20T19:41:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.876+0000 7f7ca77fe640 1 -- 172.21.15.114:0/1810203639 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f7ca8004e60 con 0x7f7cb4106800 2024-09-20T19:41:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.876+0000 7f7ca77fe640 1 --2- 172.21.15.114:0/1810203639 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7ca006e2b0 0x7f7ca0070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.877+0000 7f7ca77fe640 1 -- 172.21.15.114:0/1810203639 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f7ca808da20 con 0x7f7cb4106800 2024-09-20T19:41:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.877+0000 7f7cba4ca640 1 --2- 172.21.15.114:0/1810203639 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7ca006e2b0 0x7f7ca0070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.877+0000 7f7cba4ca640 1 --2- 172.21.15.114:0/1810203639 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7ca006e2b0 0x7f7ca0070770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0000c00 tx=0x7f7cb0031400 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:57.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:57.881+0000 7f7ca77fe640 1 -- 172.21.15.114:0/1810203639 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f7ca805b6c0 con 0x7f7cb4106800 2024-09-20T19:41:58.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:58.047+0000 7f7cbc72e640 1 -- 172.21.15.114:0/1810203639 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f7cb4109d80 con 0x7f7cb4106800 2024-09-20T19:41:58.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:58.048+0000 7f7ca77fe640 1 -- 172.21.15.114:0/1810203639 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v40) ==== 86+0+160 (secure 0 0 0) 0x7f7ca805f370 con 0x7f7cb4106800 2024-09-20T19:41:58.049 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:41:58.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:58.050+0000 7f7cbc72e640 1 -- 172.21.15.114:0/1810203639 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7ca006e2b0 msgr2=0x7f7ca0070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:58.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:58.050+0000 7f7cbc72e640 1 --2- 172.21.15.114:0/1810203639 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7ca006e2b0 0x7f7ca0070770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0000c00 tx=0x7f7cb0031400 comp rx=0 tx=0).stop 2024-09-20T19:41:58.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:58.050+0000 7f7cbc72e640 1 -- 172.21.15.114:0/1810203639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb4106800 msgr2=0x7f7cb419db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:58.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:58.050+0000 7f7cbc72e640 1 --2- 172.21.15.114:0/1810203639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb4106800 0x7f7cb419db60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800d6f0 tx=0x7f7ca800dbb0 comp rx=0 tx=0).stop 2024-09-20T19:41:58.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:58.050+0000 7f7cbc72e640 1 -- 172.21.15.114:0/1810203639 shutdown_connections 2024-09-20T19:41:58.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:58.050+0000 7f7cbc72e640 1 --2- 172.21.15.114:0/1810203639 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7ca006e2b0 0x7f7ca0070770 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:58.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:58.050+0000 7f7cbc72e640 1 --2- 172.21.15.114:0/1810203639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb4106800 0x7f7cb419db60 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:58.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:58.050+0000 7f7cbc72e640 1 --2- 172.21.15.114:0/1810203639 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7cb4100800 0x7f7cb419d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:58.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:58.050+0000 7f7cbc72e640 1 -- 172.21.15.114:0/1810203639 >> 172.21.15.114:0/1810203639 conn(0x7f7cb40fc4e0 msgr2=0x7f7cb406a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:58.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:58.050+0000 7f7cbc72e640 1 -- 172.21.15.114:0/1810203639 shutdown_connections 2024-09-20T19:41:58.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:58.050+0000 7f7cbc72e640 1 -- 172.21.15.114:0/1810203639 wait complete. 2024-09-20T19:41:58.423 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:58 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1810203639' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-20T19:41:58.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:58 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1810203639' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-20T19:41:58.721 INFO:teuthology.orchestra.run.smithi114.stdout:[{"version":40,"timestamp":"2024-09-20T19:41:47.625195+0000","name":"","changes":[{"name":"osd.5/osd_mclock_max_capacity_iops_ssd","new_value":"8115.859697"}]}] 2024-09-20T19:41:58.721 INFO:tasks.ceph_manager:config epoch is 40 2024-09-20T19:41:58.722 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-20T19:41:58.722 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-20T19:41:58.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph mgr dump --format=json 2024-09-20T19:41:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:41:59.343 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:41:59 smithi114 ceph-mon[28260]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:41:59.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.421+0000 7ff108d18640 1 -- 172.21.15.114:0/604982859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff104102b80 msgr2=0x7ff104102f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:59.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.421+0000 7ff108d18640 1 --2- 172.21.15.114:0/604982859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff104102b80 0x7ff104102f60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009c20 tx=0x7ff0f402aa00 comp rx=0 tx=0).stop 2024-09-20T19:41:59.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.421+0000 7ff108d18640 1 -- 172.21.15.114:0/604982859 shutdown_connections 2024-09-20T19:41:59.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.421+0000 7ff108d18640 1 --2- 172.21.15.114:0/604982859 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff1041034a0 0x7ff10410b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:59.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.421+0000 7ff108d18640 1 --2- 172.21.15.114:0/604982859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff104102b80 0x7ff104102f60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:59.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.421+0000 7ff108d18640 1 -- 172.21.15.114:0/604982859 >> 172.21.15.114:0/604982859 conn(0x7ff1040fc520 msgr2=0x7ff1040fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:59.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.421+0000 7ff108d18640 1 -- 172.21.15.114:0/604982859 shutdown_connections 2024-09-20T19:41:59.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.421+0000 7ff108d18640 1 -- 172.21.15.114:0/604982859 wait complete. 2024-09-20T19:41:59.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.422+0000 7ff108d18640 1 Processor -- start 2024-09-20T19:41:59.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.422+0000 7ff108d18640 1 -- start start 2024-09-20T19:41:59.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.422+0000 7ff108d18640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff104102b80 0x7ff1041007e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:59.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.422+0000 7ff108d18640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1041034a0 0x7ff104100d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:59.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.422+0000 7ff108d18640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff10410ee10 con 0x7ff1041034a0 2024-09-20T19:41:59.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.422+0000 7ff108d18640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff10410ec90 con 0x7ff104102b80 2024-09-20T19:41:59.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.422+0000 7ff101d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1041034a0 0x7ff104100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:59.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.422+0000 7ff101d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1041034a0 0x7ff104100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52852/0 (socket says 172.21.15.114:52852) 2024-09-20T19:41:59.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.422+0000 7ff101d9b640 1 -- 172.21.15.114:0/1592950200 learned_addr learned my addr 172.21.15.114:0/1592950200 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:41:59.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.422+0000 7ff10259c640 1 --2- 172.21.15.114:0/1592950200 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff104102b80 0x7ff1041007e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:59.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.422+0000 7ff101d9b640 1 -- 172.21.15.114:0/1592950200 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff104102b80 msgr2=0x7ff1041007e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:59.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.422+0000 7ff101d9b640 1 --2- 172.21.15.114:0/1592950200 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff104102b80 0x7ff1041007e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:59.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.422+0000 7ff101d9b640 1 -- 172.21.15.114:0/1592950200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff104101260 con 0x7ff1041034a0 2024-09-20T19:41:59.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.422+0000 7ff10259c640 1 --2- 172.21.15.114:0/1592950200 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff104102b80 0x7ff1041007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-20T19:41:59.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.423+0000 7ff101d9b640 1 --2- 172.21.15.114:0/1592950200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1041034a0 0x7ff104100d20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800d6f0 tx=0x7ff0f800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:59.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.423+0000 7ff0f37fe640 1 -- 172.21.15.114:0/1592950200 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff0f8017070 con 0x7ff1041034a0 2024-09-20T19:41:59.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.423+0000 7ff108d18640 1 -- 172.21.15.114:0/1592950200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff104101550 con 0x7ff1041034a0 2024-09-20T19:41:59.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.423+0000 7ff0f37fe640 1 -- 172.21.15.114:0/1592950200 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff0f800de00 con 0x7ff1041034a0 2024-09-20T19:41:59.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.423+0000 7ff0f37fe640 1 -- 172.21.15.114:0/1592950200 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff0f8004bb0 con 0x7ff1041034a0 2024-09-20T19:41:59.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.423+0000 7ff108d18640 1 -- 172.21.15.114:0/1592950200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1041a3f20 con 0x7ff1041034a0 2024-09-20T19:41:59.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.424+0000 7ff108d18640 1 -- 172.21.15.114:0/1592950200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff10406ab40 con 0x7ff1041034a0 2024-09-20T19:41:59.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.424+0000 7ff0f37fe640 1 -- 172.21.15.114:0/1592950200 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7ff0f8004e60 con 0x7ff1041034a0 2024-09-20T19:41:59.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.425+0000 7ff0f37fe640 1 --2- 172.21.15.114:0/1592950200 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff0ec06e2b0 0x7ff0ec070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:41:59.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.425+0000 7ff10259c640 1 --2- 172.21.15.114:0/1592950200 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff0ec06e2b0 0x7ff0ec070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:41:59.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.425+0000 7ff0f37fe640 1 -- 172.21.15.114:0/1592950200 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7ff0f808da90 con 0x7ff1041034a0 2024-09-20T19:41:59.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.425+0000 7ff10259c640 1 --2- 172.21.15.114:0/1592950200 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff0ec06e2b0 0x7ff0ec070770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4006010 tx=0x7ff0f402e9a0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:41:59.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.429+0000 7ff0f37fe640 1 -- 172.21.15.114:0/1592950200 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7ff0f805b730 con 0x7ff1041034a0 2024-09-20T19:41:59.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:41:59 smithi160 ceph-mon[35859]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:41:59.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.639+0000 7ff108d18640 1 -- 172.21.15.114:0/1592950200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7ff104101d10 con 0x7ff1041034a0 2024-09-20T19:41:59.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.642+0000 7ff0f37fe640 1 -- 172.21.15.114:0/1592950200 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) ==== 74+0+176572 (secure 0 0 0) 0x7ff0f805f3e0 con 0x7ff1041034a0 2024-09-20T19:41:59.645 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:41:59.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.644+0000 7ff108d18640 1 -- 172.21.15.114:0/1592950200 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff0ec06e2b0 msgr2=0x7ff0ec070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:59.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.644+0000 7ff108d18640 1 --2- 172.21.15.114:0/1592950200 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff0ec06e2b0 0x7ff0ec070770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4006010 tx=0x7ff0f402e9a0 comp rx=0 tx=0).stop 2024-09-20T19:41:59.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.644+0000 7ff108d18640 1 -- 172.21.15.114:0/1592950200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1041034a0 msgr2=0x7ff104100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:41:59.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.644+0000 7ff108d18640 1 --2- 172.21.15.114:0/1592950200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1041034a0 0x7ff104100d20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800d6f0 tx=0x7ff0f800dbb0 comp rx=0 tx=0).stop 2024-09-20T19:41:59.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.644+0000 7ff108d18640 1 -- 172.21.15.114:0/1592950200 shutdown_connections 2024-09-20T19:41:59.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.644+0000 7ff108d18640 1 --2- 172.21.15.114:0/1592950200 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff0ec06e2b0 0x7ff0ec070770 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:59.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.644+0000 7ff108d18640 1 --2- 172.21.15.114:0/1592950200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1041034a0 0x7ff104100d20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:59.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.644+0000 7ff108d18640 1 --2- 172.21.15.114:0/1592950200 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff104102b80 0x7ff1041007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:41:59.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.644+0000 7ff108d18640 1 -- 172.21.15.114:0/1592950200 >> 172.21.15.114:0/1592950200 conn(0x7ff1040fc520 msgr2=0x7ff104107ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:41:59.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.644+0000 7ff108d18640 1 -- 172.21.15.114:0/1592950200 shutdown_connections 2024-09-20T19:41:59.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:41:59.645+0000 7ff108d18640 1 -- 172.21.15.114:0/1592950200 wait complete. 2024-09-20T19:42:00.364 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:00 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1592950200' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T19:42:00.392 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":19,"flags":0,"active_gid":14219,"active_name":"smithi114.rvhgji","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":1083342253},{"type":"v1","addr":"172.21.15.114:6801","nonce":1083342253}]},"active_addr":"172.21.15.114:6801/1083342253","active_change":"2024-09-20T19:37:56.220827+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":14244,"name":"smithi160.bspiak","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.114:8443/","prometheus":"http://172.21.15.114:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":462032021}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":90469913}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":1217667147}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":1117758437}]}]} 2024-09-20T19:42:00.398 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-20T19:42:00.398 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-20T19:42:00.398 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T19:42:00.436 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:00 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1592950200' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T19:42:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:01.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.069+0000 7fb466ca3640 1 -- 172.21.15.114:0/3252180006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb460100fc0 msgr2=0x7fb460101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:01.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.069+0000 7fb466ca3640 1 --2- 172.21.15.114:0/3252180006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb460100fc0 0x7fb460101420 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb44c009880 tx=0x7fb44c02a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:01.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.069+0000 7fb466ca3640 1 -- 172.21.15.114:0/3252180006 shutdown_connections 2024-09-20T19:42:01.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.069+0000 7fb466ca3640 1 --2- 172.21.15.114:0/3252180006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb460100fc0 0x7fb460101420 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:01.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.069+0000 7fb466ca3640 1 --2- 172.21.15.114:0/3252180006 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4601057f0 0x7fb460105bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:01.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.069+0000 7fb466ca3640 1 -- 172.21.15.114:0/3252180006 >> 172.21.15.114:0/3252180006 conn(0x7fb4600fbc60 msgr2=0x7fb4600fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:01.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.069+0000 7fb466ca3640 1 -- 172.21.15.114:0/3252180006 shutdown_connections 2024-09-20T19:42:01.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.069+0000 7fb466ca3640 1 -- 172.21.15.114:0/3252180006 wait complete. 2024-09-20T19:42:01.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.070+0000 7fb466ca3640 1 Processor -- start 2024-09-20T19:42:01.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.070+0000 7fb466ca3640 1 -- start start 2024-09-20T19:42:01.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.070+0000 7fb466ca3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb460100fc0 0x7fb46019d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:01.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.070+0000 7fb466ca3640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4601057f0 0x7fb46019db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:01.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.070+0000 7fb466ca3640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb46010dd40 con 0x7fb460100fc0 2024-09-20T19:42:01.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.070+0000 7fb466ca3640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fb46010dbc0 con 0x7fb4601057f0 2024-09-20T19:42:01.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.070+0000 7fb464a3f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb460100fc0 0x7fb46019d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:01.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.070+0000 7fb464a3f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb460100fc0 0x7fb46019d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52862/0 (socket says 172.21.15.114:52862) 2024-09-20T19:42:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.070+0000 7fb464a3f640 1 -- 172.21.15.114:0/6569427 learned_addr learned my addr 172.21.15.114:0/6569427 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.071+0000 7fb464a3f640 1 -- 172.21.15.114:0/6569427 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4601057f0 msgr2=0x7fb46019db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.071+0000 7fb45ffff640 1 --2- 172.21.15.114:0/6569427 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4601057f0 0x7fb46019db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.071+0000 7fb464a3f640 1 --2- 172.21.15.114:0/6569427 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4601057f0 0x7fb46019db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.071+0000 7fb464a3f640 1 -- 172.21.15.114:0/6569427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4601977b0 con 0x7fb460100fc0 2024-09-20T19:42:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.071+0000 7fb45ffff640 1 --2- 172.21.15.114:0/6569427 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4601057f0 0x7fb46019db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-20T19:42:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.071+0000 7fb464a3f640 1 --2- 172.21.15.114:0/6569427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb460100fc0 0x7fb46019d630 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb45400b420 tx=0x7fb45400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.071+0000 7fb45dffb640 1 -- 172.21.15.114:0/6569427 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb454011630 con 0x7fb460100fc0 2024-09-20T19:42:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.071+0000 7fb45dffb640 1 -- 172.21.15.114:0/6569427 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb454016070 con 0x7fb460100fc0 2024-09-20T19:42:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.071+0000 7fb466ca3640 1 -- 172.21.15.114:0/6569427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb460197aa0 con 0x7fb460100fc0 2024-09-20T19:42:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.071+0000 7fb45dffb640 1 -- 172.21.15.114:0/6569427 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb454010d40 con 0x7fb460100fc0 2024-09-20T19:42:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.071+0000 7fb466ca3640 1 -- 172.21.15.114:0/6569427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4601980b0 con 0x7fb460100fc0 2024-09-20T19:42:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.072+0000 7fb466ca3640 1 -- 172.21.15.114:0/6569427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb460101fd0 con 0x7fb460100fc0 2024-09-20T19:42:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.073+0000 7fb45dffb640 1 -- 172.21.15.114:0/6569427 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fb45401b460 con 0x7fb460100fc0 2024-09-20T19:42:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.073+0000 7fb45dffb640 1 --2- 172.21.15.114:0/6569427 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fb45006e2b0 0x7fb450070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:01.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.073+0000 7fb45dffb640 1 -- 172.21.15.114:0/6569427 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7fb45408d680 con 0x7fb460100fc0 2024-09-20T19:42:01.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.073+0000 7fb45ffff640 1 --2- 172.21.15.114:0/6569427 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fb45006e2b0 0x7fb450070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:01.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.074+0000 7fb45ffff640 1 --2- 172.21.15.114:0/6569427 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fb45006e2b0 0x7fb450070770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb44c031460 tx=0x7fb44c0095f0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:01.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.078+0000 7fb45dffb640 1 -- 172.21.15.114:0/6569427 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fb45405b320 con 0x7fb460100fc0 2024-09-20T19:42:01.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.249+0000 7fb466ca3640 1 -- 172.21.15.114:0/6569427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb460109d60 con 0x7fb460100fc0 2024-09-20T19:42:01.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.250+0000 7fb45dffb640 1 -- 172.21.15.114:0/6569427 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+11418 (secure 0 0 0) 0x7fb45405efd0 con 0x7fb460100fc0 2024-09-20T19:42:01.251 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:42:01.251 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":41,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","created":"2024-09-20T19:35:57.433837+0000","modified":"2024-09-20T19:41:49.627858+0000","last_up_change":"2024-09-20T19:41:48.626363+0000","last_in_change":"2024-09-20T19:41:25.231165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T19:40:32.545286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9ac4e0d5-030f-4de4-96a7-fd80f6c10a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":928370375},{"type":"v1","addr":"172.21.15.114:6803","nonce":928370375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":928370375},{"type":"v1","addr":"172.21.15.114:6805","nonce":928370375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":928370375},{"type":"v1","addr":"172.21.15.114:6809","nonce":928370375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":928370375},{"type":"v1","addr":"172.21.15.114:6807","nonce":928370375}]},"public_addr":"172.21.15.114:6803/928370375","cluster_addr":"172.21.15.114:6805/928370375","heartbeat_back_addr":"172.21.15.114:6809/928370375","heartbeat_front_addr":"172.21.15.114:6807/928370375","state":["exists","up"]},{"osd":1,"uuid":"98aef96e-fc81-45ad-9cc1-49ce9ba617b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":326057805},{"type":"v1","addr":"172.21.15.114:6811","nonce":326057805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":326057805},{"type":"v1","addr":"172.21.15.114:6813","nonce":326057805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":326057805},{"type":"v1","addr":"172.21.15.114:6817","nonce":326057805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":326057805},{"type":"v1","addr":"172.21.15.114:6815","nonce":326057805}]},"public_addr":"172.21.15.114:6811/326057805","cluster_addr":"172.21.15.114:6813/326057805","heartbeat_back_addr":"172.21.15.114:6817/326057805","heartbeat_front_addr":"172.21.15.114:6815/326057805","state":["exists","up"]},{"osd":2,"uuid":"89c2fdec-88bf-4bcb-94be-fef898abddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4082826493},{"type":"v1","addr":"172.21.15.114:6819","nonce":4082826493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4082826493},{"type":"v1","addr":"172.21.15.114:6821","nonce":4082826493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4082826493},{"type":"v1","addr":"172.21.15.114:6825","nonce":4082826493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4082826493},{"type":"v1","addr":"172.21.15.114:6823","nonce":4082826493}]},"public_addr":"172.21.15.114:6819/4082826493","cluster_addr":"172.21.15.114:6821/4082826493","heartbeat_back_addr":"172.21.15.114:6825/4082826493","heartbeat_front_addr":"172.21.15.114:6823/4082826493","state":["exists","up"]},{"osd":3,"uuid":"64f8a718-5814-4210-8696-b2acd153c046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":496816402},{"type":"v1","addr":"172.21.15.160:6801","nonce":496816402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":496816402},{"type":"v1","addr":"172.21.15.160:6803","nonce":496816402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":496816402},{"type":"v1","addr":"172.21.15.160:6807","nonce":496816402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":496816402},{"type":"v1","addr":"172.21.15.160:6805","nonce":496816402}]},"public_addr":"172.21.15.160:6801/496816402","cluster_addr":"172.21.15.160:6803/496816402","heartbeat_back_addr":"172.21.15.160:6807/496816402","heartbeat_front_addr":"172.21.15.160:6805/496816402","state":["exists","up"]},{"osd":4,"uuid":"302c7425-1a84-44fc-a4c9-58a71a9cdc54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3464010154},{"type":"v1","addr":"172.21.15.160:6809","nonce":3464010154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3464010154},{"type":"v1","addr":"172.21.15.160:6811","nonce":3464010154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3464010154},{"type":"v1","addr":"172.21.15.160:6815","nonce":3464010154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3464010154},{"type":"v1","addr":"172.21.15.160:6813","nonce":3464010154}]},"public_addr":"172.21.15.160:6809/3464010154","cluster_addr":"172.21.15.160:6811/3464010154","heartbeat_back_addr":"172.21.15.160:6815/3464010154","heartbeat_front_addr":"172.21.15.160:6813/3464010154","state":["exists","up"]},{"osd":5,"uuid":"a645a156-dc73-4033-adde-4d2380a9c000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":3716955701},{"type":"v1","addr":"172.21.15.160:6817","nonce":3716955701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":3716955701},{"type":"v1","addr":"172.21.15.160:6819","nonce":3716955701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":3716955701},{"type":"v1","addr":"172.21.15.160:6823","nonce":3716955701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":3716955701},{"type":"v1","addr":"172.21.15.160:6821","nonce":3716955701}]},"public_addr":"172.21.15.160:6817/3716955701","cluster_addr":"172.21.15.160:6819/3716955701","heartbeat_back_addr":"172.21.15.160:6823/3716955701","heartbeat_front_addr":"172.21.15.160:6821/3716955701","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:39:31.890087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:39:59.898001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:40:28.463295+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:40:53.291600+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:41:20.621298+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:41:46.675541+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/1191801334":"2024-09-21T19:37:56.220717+0000","172.21.15.114:0/3994747756":"2024-09-21T19:37:56.220717+0000","172.21.15.114:0/3541238036":"2024-09-21T19:36:27.357221+0000","172.21.15.114:0/62878745":"2024-09-21T19:36:27.357221+0000","172.21.15.114:0/4279612706":"2024-09-21T19:36:27.357221+0000","172.21.15.114:0/2669286306":"2024-09-21T19:36:58.132711+0000","172.21.15.114:6800/689537755":"2024-09-21T19:36:27.357221+0000","172.21.15.114:6801/689537755":"2024-09-21T19:36:27.357221+0000","172.21.15.114:6801/2644929394":"2024-09-21T19:36:58.132711+0000","172.21.15.114:0/1230605596":"2024-09-21T19:36:58.132711+0000","172.21.15.114:6800/969843775":"2024-09-21T19:37:56.220717+0000","172.21.15.114:6800/2644929394":"2024-09-21T19:36:58.132711+0000","172.21.15.114:0/2896121214":"2024-09-21T19:36:58.132711+0000","172.21.15.114:6801/969843775":"2024-09-21T19:37:56.220717+0000","172.21.15.114:0/2774428689":"2024-09-21T19:37:56.220717+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T19:42:01.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.252+0000 7fb466ca3640 1 -- 172.21.15.114:0/6569427 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fb45006e2b0 msgr2=0x7fb450070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:01.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.252+0000 7fb466ca3640 1 --2- 172.21.15.114:0/6569427 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fb45006e2b0 0x7fb450070770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb44c031460 tx=0x7fb44c0095f0 comp rx=0 tx=0).stop 2024-09-20T19:42:01.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.252+0000 7fb466ca3640 1 -- 172.21.15.114:0/6569427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb460100fc0 msgr2=0x7fb46019d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:01.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.252+0000 7fb466ca3640 1 --2- 172.21.15.114:0/6569427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb460100fc0 0x7fb46019d630 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb45400b420 tx=0x7fb45400b8e0 comp rx=0 tx=0).stop 2024-09-20T19:42:01.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.252+0000 7fb466ca3640 1 -- 172.21.15.114:0/6569427 shutdown_connections 2024-09-20T19:42:01.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.252+0000 7fb466ca3640 1 --2- 172.21.15.114:0/6569427 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fb45006e2b0 0x7fb450070770 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:01.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.252+0000 7fb466ca3640 1 --2- 172.21.15.114:0/6569427 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4601057f0 0x7fb46019db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:01.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.252+0000 7fb466ca3640 1 --2- 172.21.15.114:0/6569427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb460100fc0 0x7fb46019d630 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:01.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.252+0000 7fb466ca3640 1 -- 172.21.15.114:0/6569427 >> 172.21.15.114:0/6569427 conn(0x7fb4600fbc60 msgr2=0x7fb460100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:01.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.252+0000 7fb466ca3640 1 -- 172.21.15.114:0/6569427 shutdown_connections 2024-09-20T19:42:01.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:01.252+0000 7fb466ca3640 1 -- 172.21.15.114:0/6569427 wait complete. 2024-09-20T19:42:01.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:01 smithi114 ceph-mon[28260]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:01.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:01 smithi160 ceph-mon[35859]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:01.839 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-20T19:42:01.839 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T19:42:02.085 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:02.365 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:02 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/6569427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T19:42:02.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:02 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/6569427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T19:42:02.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.550+0000 7f4dd5a30640 1 -- 172.21.15.114:0/2496950482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dd0101930 msgr2=0x7f4dd0101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:02.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.550+0000 7f4dd5a30640 1 --2- 172.21.15.114:0/2496950482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dd0101930 0x7f4dd0101d90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4009880 tx=0x7f4dc402a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:02.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.550+0000 7f4dd5a30640 1 -- 172.21.15.114:0/2496950482 shutdown_connections 2024-09-20T19:42:02.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.550+0000 7f4dd5a30640 1 --2- 172.21.15.114:0/2496950482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dd0101930 0x7f4dd0101d90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:02.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.550+0000 7f4dd5a30640 1 --2- 172.21.15.114:0/2496950482 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4dd0107930 0x7f4dd0107d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:02.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.550+0000 7f4dd5a30640 1 -- 172.21.15.114:0/2496950482 >> 172.21.15.114:0/2496950482 conn(0x7f4dd00fd630 msgr2=0x7f4dd00ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:02.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.550+0000 7f4dd5a30640 1 -- 172.21.15.114:0/2496950482 shutdown_connections 2024-09-20T19:42:02.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.550+0000 7f4dd5a30640 1 -- 172.21.15.114:0/2496950482 wait complete. 2024-09-20T19:42:02.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.550+0000 7f4dd5a30640 1 Processor -- start 2024-09-20T19:42:02.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dd5a30640 1 -- start start 2024-09-20T19:42:02.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dd5a30640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dd0101930 0x7f4dd00744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:02.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dd5a30640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4dd0107930 0x7f4dd0074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:02.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dd5a30640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd010ee60 con 0x7f4dd0101930 2024-09-20T19:42:02.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dd5a30640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd010ece0 con 0x7f4dd0107930 2024-09-20T19:42:02.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dceffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dd0101930 0x7f4dd00744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:02.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dceffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dd0101930 0x7f4dd00744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52890/0 (socket says 172.21.15.114:52890) 2024-09-20T19:42:02.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dceffd640 1 -- 172.21.15.114:0/3079990258 learned_addr learned my addr 172.21.15.114:0/3079990258 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:02.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dce7fc640 1 --2- 172.21.15.114:0/3079990258 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4dd0107930 0x7f4dd0074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:02.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dceffd640 1 -- 172.21.15.114:0/3079990258 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4dd0107930 msgr2=0x7f4dd0074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:02.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dceffd640 1 --2- 172.21.15.114:0/3079990258 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4dd0107930 0x7f4dd0074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:02.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dceffd640 1 -- 172.21.15.114:0/3079990258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd0074f70 con 0x7f4dd0101930 2024-09-20T19:42:02.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dceffd640 1 --2- 172.21.15.114:0/3079990258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dd0101930 0x7f4dd00744f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4dc000b420 tx=0x7f4dc000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:02.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dd4a2e640 1 -- 172.21.15.114:0/3079990258 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f4dc0011630 con 0x7f4dd0101930 2024-09-20T19:42:02.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dd5a30640 1 -- 172.21.15.114:0/3079990258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dd0075260 con 0x7f4dd0101930 2024-09-20T19:42:02.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.551+0000 7f4dd4a2e640 1 -- 172.21.15.114:0/3079990258 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4dc0016070 con 0x7f4dd0101930 2024-09-20T19:42:02.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.552+0000 7f4dd4a2e640 1 -- 172.21.15.114:0/3079990258 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f4dc0010d40 con 0x7f4dd0101930 2024-09-20T19:42:02.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.552+0000 7f4dd5a30640 1 -- 172.21.15.114:0/3079990258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dd01a3ef0 con 0x7f4dd0101930 2024-09-20T19:42:02.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.553+0000 7f4dd5a30640 1 -- 172.21.15.114:0/3079990258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dd0103070 con 0x7f4dd0101930 2024-09-20T19:42:02.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.553+0000 7f4dd4a2e640 1 -- 172.21.15.114:0/3079990258 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f4dc001b460 con 0x7f4dd0101930 2024-09-20T19:42:02.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.554+0000 7f4dd4a2e640 1 --2- 172.21.15.114:0/3079990258 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f4db806e2b0 0x7f4db8070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:02.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.554+0000 7f4dce7fc640 1 --2- 172.21.15.114:0/3079990258 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f4db806e2b0 0x7f4db8070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:02.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.554+0000 7f4dd4a2e640 1 -- 172.21.15.114:0/3079990258 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f4dc008d740 con 0x7f4dd0101930 2024-09-20T19:42:02.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.554+0000 7f4dce7fc640 1 --2- 172.21.15.114:0/3079990258 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f4db806e2b0 0x7f4db8070770 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0075f80 tx=0x7f4dc4031400 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:02.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.558+0000 7f4dd4a2e640 1 -- 172.21.15.114:0/3079990258 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f4dc005b380 con 0x7f4dd0101930 2024-09-20T19:42:02.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.726+0000 7f4dd5a30640 1 -- 172.21.15.114:0/3079990258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4dd0109ec0 con 0x7f4dd0101930 2024-09-20T19:42:02.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.727+0000 7f4dd4a2e640 1 -- 172.21.15.114:0/3079990258 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+11418 (secure 0 0 0) 0x7f4dc005f030 con 0x7f4dd0101930 2024-09-20T19:42:02.729 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:42:02.729 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":41,"fsid":"6a49c7bc-7787-11ef-baf5-efdc52797490","created":"2024-09-20T19:35:57.433837+0000","modified":"2024-09-20T19:41:49.627858+0000","last_up_change":"2024-09-20T19:41:48.626363+0000","last_in_change":"2024-09-20T19:41:25.231165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T19:40:32.545286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9ac4e0d5-030f-4de4-96a7-fd80f6c10a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":928370375},{"type":"v1","addr":"172.21.15.114:6803","nonce":928370375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":928370375},{"type":"v1","addr":"172.21.15.114:6805","nonce":928370375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":928370375},{"type":"v1","addr":"172.21.15.114:6809","nonce":928370375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":928370375},{"type":"v1","addr":"172.21.15.114:6807","nonce":928370375}]},"public_addr":"172.21.15.114:6803/928370375","cluster_addr":"172.21.15.114:6805/928370375","heartbeat_back_addr":"172.21.15.114:6809/928370375","heartbeat_front_addr":"172.21.15.114:6807/928370375","state":["exists","up"]},{"osd":1,"uuid":"98aef96e-fc81-45ad-9cc1-49ce9ba617b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":326057805},{"type":"v1","addr":"172.21.15.114:6811","nonce":326057805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":326057805},{"type":"v1","addr":"172.21.15.114:6813","nonce":326057805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":326057805},{"type":"v1","addr":"172.21.15.114:6817","nonce":326057805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":326057805},{"type":"v1","addr":"172.21.15.114:6815","nonce":326057805}]},"public_addr":"172.21.15.114:6811/326057805","cluster_addr":"172.21.15.114:6813/326057805","heartbeat_back_addr":"172.21.15.114:6817/326057805","heartbeat_front_addr":"172.21.15.114:6815/326057805","state":["exists","up"]},{"osd":2,"uuid":"89c2fdec-88bf-4bcb-94be-fef898abddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4082826493},{"type":"v1","addr":"172.21.15.114:6819","nonce":4082826493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4082826493},{"type":"v1","addr":"172.21.15.114:6821","nonce":4082826493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4082826493},{"type":"v1","addr":"172.21.15.114:6825","nonce":4082826493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4082826493},{"type":"v1","addr":"172.21.15.114:6823","nonce":4082826493}]},"public_addr":"172.21.15.114:6819/4082826493","cluster_addr":"172.21.15.114:6821/4082826493","heartbeat_back_addr":"172.21.15.114:6825/4082826493","heartbeat_front_addr":"172.21.15.114:6823/4082826493","state":["exists","up"]},{"osd":3,"uuid":"64f8a718-5814-4210-8696-b2acd153c046","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":496816402},{"type":"v1","addr":"172.21.15.160:6801","nonce":496816402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":496816402},{"type":"v1","addr":"172.21.15.160:6803","nonce":496816402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":496816402},{"type":"v1","addr":"172.21.15.160:6807","nonce":496816402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":496816402},{"type":"v1","addr":"172.21.15.160:6805","nonce":496816402}]},"public_addr":"172.21.15.160:6801/496816402","cluster_addr":"172.21.15.160:6803/496816402","heartbeat_back_addr":"172.21.15.160:6807/496816402","heartbeat_front_addr":"172.21.15.160:6805/496816402","state":["exists","up"]},{"osd":4,"uuid":"302c7425-1a84-44fc-a4c9-58a71a9cdc54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3464010154},{"type":"v1","addr":"172.21.15.160:6809","nonce":3464010154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3464010154},{"type":"v1","addr":"172.21.15.160:6811","nonce":3464010154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3464010154},{"type":"v1","addr":"172.21.15.160:6815","nonce":3464010154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3464010154},{"type":"v1","addr":"172.21.15.160:6813","nonce":3464010154}]},"public_addr":"172.21.15.160:6809/3464010154","cluster_addr":"172.21.15.160:6811/3464010154","heartbeat_back_addr":"172.21.15.160:6815/3464010154","heartbeat_front_addr":"172.21.15.160:6813/3464010154","state":["exists","up"]},{"osd":5,"uuid":"a645a156-dc73-4033-adde-4d2380a9c000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":3716955701},{"type":"v1","addr":"172.21.15.160:6817","nonce":3716955701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":3716955701},{"type":"v1","addr":"172.21.15.160:6819","nonce":3716955701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":3716955701},{"type":"v1","addr":"172.21.15.160:6823","nonce":3716955701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":3716955701},{"type":"v1","addr":"172.21.15.160:6821","nonce":3716955701}]},"public_addr":"172.21.15.160:6817/3716955701","cluster_addr":"172.21.15.160:6819/3716955701","heartbeat_back_addr":"172.21.15.160:6823/3716955701","heartbeat_front_addr":"172.21.15.160:6821/3716955701","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:39:31.890087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:39:59.898001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:40:28.463295+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:40:53.291600+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:41:20.621298+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T19:41:46.675541+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/1191801334":"2024-09-21T19:37:56.220717+0000","172.21.15.114:0/3994747756":"2024-09-21T19:37:56.220717+0000","172.21.15.114:0/3541238036":"2024-09-21T19:36:27.357221+0000","172.21.15.114:0/62878745":"2024-09-21T19:36:27.357221+0000","172.21.15.114:0/4279612706":"2024-09-21T19:36:27.357221+0000","172.21.15.114:0/2669286306":"2024-09-21T19:36:58.132711+0000","172.21.15.114:6800/689537755":"2024-09-21T19:36:27.357221+0000","172.21.15.114:6801/689537755":"2024-09-21T19:36:27.357221+0000","172.21.15.114:6801/2644929394":"2024-09-21T19:36:58.132711+0000","172.21.15.114:0/1230605596":"2024-09-21T19:36:58.132711+0000","172.21.15.114:6800/969843775":"2024-09-21T19:37:56.220717+0000","172.21.15.114:6800/2644929394":"2024-09-21T19:36:58.132711+0000","172.21.15.114:0/2896121214":"2024-09-21T19:36:58.132711+0000","172.21.15.114:6801/969843775":"2024-09-21T19:37:56.220717+0000","172.21.15.114:0/2774428689":"2024-09-21T19:37:56.220717+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T19:42:02.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.730+0000 7f4dd5a30640 1 -- 172.21.15.114:0/3079990258 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f4db806e2b0 msgr2=0x7f4db8070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:02.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.730+0000 7f4dd5a30640 1 --2- 172.21.15.114:0/3079990258 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f4db806e2b0 0x7f4db8070770 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0075f80 tx=0x7f4dc4031400 comp rx=0 tx=0).stop 2024-09-20T19:42:02.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.730+0000 7f4dd5a30640 1 -- 172.21.15.114:0/3079990258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dd0101930 msgr2=0x7f4dd00744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:02.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.730+0000 7f4dd5a30640 1 --2- 172.21.15.114:0/3079990258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dd0101930 0x7f4dd00744f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4dc000b420 tx=0x7f4dc000b8e0 comp rx=0 tx=0).stop 2024-09-20T19:42:02.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.730+0000 7f4dd5a30640 1 -- 172.21.15.114:0/3079990258 shutdown_connections 2024-09-20T19:42:02.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.730+0000 7f4dd5a30640 1 --2- 172.21.15.114:0/3079990258 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f4db806e2b0 0x7f4db8070770 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:02.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.730+0000 7f4dd5a30640 1 --2- 172.21.15.114:0/3079990258 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4dd0107930 0x7f4dd0074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:02.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.730+0000 7f4dd5a30640 1 --2- 172.21.15.114:0/3079990258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dd0101930 0x7f4dd00744f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:02.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.730+0000 7f4dd5a30640 1 -- 172.21.15.114:0/3079990258 >> 172.21.15.114:0/3079990258 conn(0x7f4dd00fd630 msgr2=0x7f4dd00fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:02.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.730+0000 7f4dd5a30640 1 -- 172.21.15.114:0/3079990258 shutdown_connections 2024-09-20T19:42:02.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:02.730+0000 7f4dd5a30640 1 -- 172.21.15.114:0/3079990258 wait complete. 2024-09-20T19:42:03.410 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:03 smithi114 ceph-mon[28260]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:03.410 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:03 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3079990258' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T19:42:03.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-20T19:42:03.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-20T19:42:03.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-20T19:42:03.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-20T19:42:03.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-20T19:42:03.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-20T19:42:03.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:03 smithi160 ceph-mon[35859]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:03.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:03 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/3079990258' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T19:42:03.763 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:03.790 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:04.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.594+0000 7f2bd244b640 1 -- 172.21.15.114:0/317043883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc106810 msgr2=0x7f2bcc106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:04.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.594+0000 7f2bd244b640 1 --2- 172.21.15.114:0/317043883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc106810 0x7f2bcc106bf0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009880 tx=0x7f2bbc02a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:04.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.594+0000 7f2bd244b640 1 -- 172.21.15.114:0/317043883 shutdown_connections 2024-09-20T19:42:04.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.594+0000 7f2bd244b640 1 --2- 172.21.15.114:0/317043883 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2bcc100810 0x7f2bcc100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:04.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.594+0000 7f2bd244b640 1 --2- 172.21.15.114:0/317043883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc106810 0x7f2bcc106bf0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:04.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.594+0000 7f2bd244b640 1 -- 172.21.15.114:0/317043883 >> 172.21.15.114:0/317043883 conn(0x7f2bcc0fc510 msgr2=0x7f2bcc0fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:04.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.594+0000 7f2bd244b640 1 -- 172.21.15.114:0/317043883 shutdown_connections 2024-09-20T19:42:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.594+0000 7f2bd244b640 1 -- 172.21.15.114:0/317043883 wait complete. 2024-09-20T19:42:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.595+0000 7f2bd244b640 1 Processor -- start 2024-09-20T19:42:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.595+0000 7f2bd244b640 1 -- start start 2024-09-20T19:42:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.595+0000 7f2bd244b640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2bcc100810 0x7f2bcc19c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.595+0000 7f2bd244b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc106810 0x7f2bcc19cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.595+0000 7f2bd244b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc10dcc0 con 0x7f2bcc106810 2024-09-20T19:42:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.595+0000 7f2bd244b640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc10db40 con 0x7f2bcc100810 2024-09-20T19:42:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.595+0000 7f2bc3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc106810 0x7f2bcc19cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.595+0000 7f2bc3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc106810 0x7f2bcc19cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52902/0 (socket says 172.21.15.114:52902) 2024-09-20T19:42:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.595+0000 7f2bc3fff640 1 -- 172.21.15.114:0/1735437804 learned_addr learned my addr 172.21.15.114:0/1735437804 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.595+0000 7f2bcbfff640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2bcc100810 0x7f2bcc19c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.596+0000 7f2bc3fff640 1 -- 172.21.15.114:0/1735437804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2bcc100810 msgr2=0x7f2bcc19c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.596+0000 7f2bc3fff640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2bcc100810 0x7f2bcc19c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.596+0000 7f2bc3fff640 1 -- 172.21.15.114:0/1735437804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bcc1966b0 con 0x7f2bcc106810 2024-09-20T19:42:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.596+0000 7f2bc3fff640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc106810 0x7f2bcc19cb20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2bb400c8a0 tx=0x7f2bb400cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.596+0000 7f2bc9ffb640 1 -- 172.21.15.114:0/1735437804 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2bb40079b0 con 0x7f2bcc106810 2024-09-20T19:42:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.596+0000 7f2bc9ffb640 1 -- 172.21.15.114:0/1735437804 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2bb4007b50 con 0x7f2bcc106810 2024-09-20T19:42:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.596+0000 7f2bd244b640 1 -- 172.21.15.114:0/1735437804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bcc1969a0 con 0x7f2bcc106810 2024-09-20T19:42:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.596+0000 7f2bc9ffb640 1 -- 172.21.15.114:0/1735437804 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2bb4018700 con 0x7f2bcc106810 2024-09-20T19:42:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.596+0000 7f2bd244b640 1 -- 172.21.15.114:0/1735437804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bcc196f30 con 0x7f2bcc106810 2024-09-20T19:42:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.597+0000 7f2bd244b640 1 -- 172.21.15.114:0/1735437804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2bcc197550 con 0x7f2bcc106810 2024-09-20T19:42:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.598+0000 7f2bc9ffb640 1 -- 172.21.15.114:0/1735437804 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f2bb4007d00 con 0x7f2bcc106810 2024-09-20T19:42:04.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.598+0000 7f2bc9ffb640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2bb806e310 0x7f2bb80707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:04.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.598+0000 7f2bc9ffb640 1 -- 172.21.15.114:0/1735437804 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f2bb408dfa0 con 0x7f2bcc106810 2024-09-20T19:42:04.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.598+0000 7f2bcbfff640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2bb806e310 0x7f2bb80707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:04.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.598+0000 7f2bc9ffb640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.160:6816/3716955701,v1:172.21.15.160:6817/3716955701] conn(0x7f2bb8074100 0x7f2bb8076520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:04.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.598+0000 7f2bc9ffb640 1 -- 172.21.15.114:0/1735437804 --> [v2:172.21.15.160:6816/3716955701,v1:172.21.15.160:6817/3716955701] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2bb4012ec0 con 0x7f2bb8074100 2024-09-20T19:42:04.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.598+0000 7f2bc9ffb640 1 -- 172.21.15.114:0/1735437804 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f2bb4095050 con 0x7f2bcc106810 2024-09-20T19:42:04.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.599+0000 7f2bcbfff640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2bb806e310 0x7f2bb80707d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc000c00 tx=0x7f2bbc031400 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:04.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.599+0000 7f2bd09e8640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.160:6816/3716955701,v1:172.21.15.160:6817/3716955701] conn(0x7f2bb8074100 0x7f2bb8076520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:04.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.599+0000 7f2bd09e8640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.160:6816/3716955701,v1:172.21.15.160:6817/3716955701] conn(0x7f2bb8074100 0x7f2bb8076520 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:04.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.600+0000 7f2bc9ffb640 1 -- 172.21.15.114:0/1735437804 <== osd.5 v2:172.21.15.160:6816/3716955701 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2bb4012ec0 con 0x7f2bb8074100 2024-09-20T19:42:04.614 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:04.615 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:04.616 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:04.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.620+0000 7f2bd244b640 1 -- 172.21.15.114:0/1735437804 --> [v2:172.21.15.160:6816/3716955701,v1:172.21.15.160:6817/3716955701] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2bcc062900 con 0x7f2bb8074100 2024-09-20T19:42:04.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.620+0000 7f2bc9ffb640 1 -- 172.21.15.114:0/1735437804 <== osd.5 v2:172.21.15.160:6816/3716955701 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2bcc062900 con 0x7f2bb8074100 2024-09-20T19:42:04.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.621+0000 7f2bd244b640 1 -- 172.21.15.114:0/1735437804 >> [v2:172.21.15.160:6816/3716955701,v1:172.21.15.160:6817/3716955701] conn(0x7f2bb8074100 msgr2=0x7f2bb8076520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:04.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.621+0000 7f2bd244b640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.160:6816/3716955701,v1:172.21.15.160:6817/3716955701] conn(0x7f2bb8074100 0x7f2bb8076520 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:04.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.621+0000 7f2bd244b640 1 -- 172.21.15.114:0/1735437804 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2bb806e310 msgr2=0x7f2bb80707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:04.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.621+0000 7f2bd244b640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2bb806e310 0x7f2bb80707d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc000c00 tx=0x7f2bbc031400 comp rx=0 tx=0).stop 2024-09-20T19:42:04.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.621+0000 7f2bd244b640 1 -- 172.21.15.114:0/1735437804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc106810 msgr2=0x7f2bcc19cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:04.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.621+0000 7f2bd244b640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc106810 0x7f2bcc19cb20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2bb400c8a0 tx=0x7f2bb400cd60 comp rx=0 tx=0).stop 2024-09-20T19:42:04.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.621+0000 7f2bd244b640 1 -- 172.21.15.114:0/1735437804 shutdown_connections 2024-09-20T19:42:04.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.621+0000 7f2bd244b640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.160:6816/3716955701,v1:172.21.15.160:6817/3716955701] conn(0x7f2bb8074100 0x7f2bb8076520 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:04.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.621+0000 7f2bd244b640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2bb806e310 0x7f2bb80707d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:04.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.621+0000 7f2bd244b640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc106810 0x7f2bcc19cb20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:04.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.621+0000 7f2bd244b640 1 --2- 172.21.15.114:0/1735437804 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2bcc100810 0x7f2bcc19c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:04.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.621+0000 7f2bd244b640 1 -- 172.21.15.114:0/1735437804 >> 172.21.15.114:0/1735437804 conn(0x7f2bcc0fc510 msgr2=0x7f2bcc0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:04.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.621+0000 7f2bd244b640 1 -- 172.21.15.114:0/1735437804 shutdown_connections 2024-09-20T19:42:04.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:04.621+0000 7f2bd244b640 1 -- 172.21.15.114:0/1735437804 wait complete. 2024-09-20T19:42:05.439 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:05 smithi114 ceph-mon[28260]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:05.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.726+0000 7fee7caaa640 1 -- 172.21.15.114:0/1599223468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee78106820 msgr2=0x7fee78106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:05.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.726+0000 7fee7caaa640 1 --2- 172.21.15.114:0/1599223468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee78106820 0x7fee78106c00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fee68009880 tx=0x7fee6802a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:05.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.726+0000 7fee7caaa640 1 -- 172.21.15.114:0/1599223468 shutdown_connections 2024-09-20T19:42:05.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.726+0000 7fee7caaa640 1 --2- 172.21.15.114:0/1599223468 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fee78100820 0x7fee78100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:05.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.726+0000 7fee7caaa640 1 --2- 172.21.15.114:0/1599223468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee78106820 0x7fee78106c00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:05.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.726+0000 7fee7caaa640 1 -- 172.21.15.114:0/1599223468 >> 172.21.15.114:0/1599223468 conn(0x7fee780fc520 msgr2=0x7fee780fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:05.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.726+0000 7fee7caaa640 1 -- 172.21.15.114:0/1599223468 shutdown_connections 2024-09-20T19:42:05.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.726+0000 7fee7caaa640 1 -- 172.21.15.114:0/1599223468 wait complete. 2024-09-20T19:42:05.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.727+0000 7fee7caaa640 1 Processor -- start 2024-09-20T19:42:05.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.727+0000 7fee7caaa640 1 -- start start 2024-09-20T19:42:05.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.727+0000 7fee7caaa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee78100820 0x7fee7819c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:05.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.727+0000 7fee7caaa640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fee78106820 0x7fee7819cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:05.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.727+0000 7fee7caaa640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fee7810dcd0 con 0x7fee78100820 2024-09-20T19:42:05.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.727+0000 7fee7caaa640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fee7810db50 con 0x7fee78106820 2024-09-20T19:42:05.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.727+0000 7fee7659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee78100820 0x7fee7819c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:05.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.727+0000 7fee7659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee78100820 0x7fee7819c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57704/0 (socket says 172.21.15.114:57704) 2024-09-20T19:42:05.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.727+0000 7fee7659c640 1 -- 172.21.15.114:0/3292484233 learned_addr learned my addr 172.21.15.114:0/3292484233 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:05.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.728+0000 7fee75d9b640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fee78106820 0x7fee7819cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:05.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.728+0000 7fee7659c640 1 -- 172.21.15.114:0/3292484233 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fee78106820 msgr2=0x7fee7819cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:05.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.728+0000 7fee7659c640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fee78106820 0x7fee7819cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:05.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.728+0000 7fee7659c640 1 -- 172.21.15.114:0/3292484233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee781966f0 con 0x7fee78100820 2024-09-20T19:42:05.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.728+0000 7fee75d9b640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fee78106820 0x7fee7819cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-20T19:42:05.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.728+0000 7fee7659c640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee78100820 0x7fee7819c620 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fee68031460 tx=0x7fee680048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:05.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.728+0000 7fee677fe640 1 -- 172.21.15.114:0/3292484233 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fee6803f070 con 0x7fee78100820 2024-09-20T19:42:05.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.728+0000 7fee7caaa640 1 -- 172.21.15.114:0/3292484233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee78196980 con 0x7fee78100820 2024-09-20T19:42:05.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.728+0000 7fee677fe640 1 -- 172.21.15.114:0/3292484233 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fee68004cf0 con 0x7fee78100820 2024-09-20T19:42:05.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.728+0000 7fee677fe640 1 -- 172.21.15.114:0/3292484233 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fee68035030 con 0x7fee78100820 2024-09-20T19:42:05.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.728+0000 7fee7caaa640 1 -- 172.21.15.114:0/3292484233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee78196e80 con 0x7fee78100820 2024-09-20T19:42:05.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.729+0000 7fee7caaa640 1 -- 172.21.15.114:0/3292484233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fee781973c0 con 0x7fee78100820 2024-09-20T19:42:05.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.730+0000 7fee677fe640 1 -- 172.21.15.114:0/3292484233 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fee68042070 con 0x7fee78100820 2024-09-20T19:42:05.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.730+0000 7fee677fe640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fee6006e2b0 0x7fee60070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:05.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.730+0000 7fee677fe640 1 -- 172.21.15.114:0/3292484233 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7fee680aef30 con 0x7fee78100820 2024-09-20T19:42:05.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.730+0000 7fee75d9b640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fee6006e2b0 0x7fee60070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:05.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.730+0000 7fee677fe640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375] conn(0x7fee600740a0 0x7fee600764c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:05.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.730+0000 7fee677fe640 1 -- 172.21.15.114:0/3292484233 --> [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee68032e50 con 0x7fee600740a0 2024-09-20T19:42:05.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.730+0000 7fee677fe640 1 -- 172.21.15.114:0/3292484233 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fee680af3d0 con 0x7fee78100820 2024-09-20T19:42:05.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.730+0000 7fee76d9d640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375] conn(0x7fee600740a0 0x7fee600764c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:05.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.731+0000 7fee75d9b640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fee6006e2b0 0x7fee60070770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fee78197f40 tx=0x7fee6c005d30 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:05.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.731+0000 7fee76d9d640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375] conn(0x7fee600740a0 0x7fee600764c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:05.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.732+0000 7fee677fe640 1 -- 172.21.15.114:0/3292484233 <== osd.0 v2:172.21.15.114:6802/928370375 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee68032e50 con 0x7fee600740a0 2024-09-20T19:42:05.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee7caaa640 1 -- 172.21.15.114:0/3292484233 --> [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fee78062880 con 0x7fee600740a0 2024-09-20T19:42:05.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee677fe640 1 -- 172.21.15.114:0/3292484233 <== osd.0 v2:172.21.15.114:6802/928370375 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fee78062880 con 0x7fee600740a0 2024-09-20T19:42:05.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee7caaa640 1 -- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375] conn(0x7fee600740a0 msgr2=0x7fee600764c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:05.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee7caaa640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375] conn(0x7fee600740a0 0x7fee600764c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:05.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee7caaa640 1 -- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fee6006e2b0 msgr2=0x7fee60070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:05.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee7caaa640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fee6006e2b0 0x7fee60070770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fee78197f40 tx=0x7fee6c005d30 comp rx=0 tx=0).stop 2024-09-20T19:42:05.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee7caaa640 1 -- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee78100820 msgr2=0x7fee7819c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:05.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee7caaa640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee78100820 0x7fee7819c620 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fee68031460 tx=0x7fee680048f0 comp rx=0 tx=0).stop 2024-09-20T19:42:05.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee7caaa640 1 -- 172.21.15.114:0/3292484233 shutdown_connections 2024-09-20T19:42:05.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee7caaa640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:6802/928370375,v1:172.21.15.114:6803/928370375] conn(0x7fee600740a0 0x7fee600764c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:05.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee7caaa640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fee6006e2b0 0x7fee60070770 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:05.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee7caaa640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fee78106820 0x7fee7819cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:05.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee7caaa640 1 --2- 172.21.15.114:0/3292484233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee78100820 0x7fee7819c620 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:05.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee7caaa640 1 -- 172.21.15.114:0/3292484233 >> 172.21.15.114:0/3292484233 conn(0x7fee780fc520 msgr2=0x7fee780fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:05.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee7caaa640 1 -- 172.21.15.114:0/3292484233 shutdown_connections 2024-09-20T19:42:05.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:05.746+0000 7fee7caaa640 1 -- 172.21.15.114:0/3292484233 wait complete. 2024-09-20T19:42:05.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:05 smithi160 ceph-mon[35859]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:06.623 INFO:teuthology.orchestra.run.smithi114.stdout:38654705698 2024-09-20T19:42:06.624 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-20T19:42:06.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.801+0000 7fbb46989640 1 -- 172.21.15.114:0/3596037078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb40103460 msgr2=0x7fbb4010b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:06.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.801+0000 7fbb46989640 1 --2- 172.21.15.114:0/3596037078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb40103460 0x7fbb4010b950 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbb34009880 tx=0x7fbb3402a530 comp rx=0 tx=0).stop 2024-09-20T19:42:06.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.801+0000 7fbb46989640 1 -- 172.21.15.114:0/3596037078 shutdown_connections 2024-09-20T19:42:06.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.801+0000 7fbb46989640 1 --2- 172.21.15.114:0/3596037078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb40103460 0x7fbb4010b950 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:06.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.801+0000 7fbb46989640 1 --2- 172.21.15.114:0/3596037078 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb40102b40 0x7fbb40102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:06.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.801+0000 7fbb46989640 1 -- 172.21.15.114:0/3596037078 >> 172.21.15.114:0/3596037078 conn(0x7fbb400fc4c0 msgr2=0x7fbb400fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:06.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.801+0000 7fbb46989640 1 -- 172.21.15.114:0/3596037078 shutdown_connections 2024-09-20T19:42:06.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.801+0000 7fbb46989640 1 -- 172.21.15.114:0/3596037078 wait complete. 2024-09-20T19:42:06.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.802+0000 7fbb46989640 1 Processor -- start 2024-09-20T19:42:06.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.802+0000 7fbb46989640 1 -- start start 2024-09-20T19:42:06.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.802+0000 7fbb46989640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb40102b40 0x7fbb4019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:06.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.802+0000 7fbb46989640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb40103460 0x7fbb4019db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:06.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.802+0000 7fbb46989640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4010edb0 con 0x7fbb40102b40 2024-09-20T19:42:06.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.802+0000 7fbb46989640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4010ec30 con 0x7fbb40103460 2024-09-20T19:42:06.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.802+0000 7fbb3ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb40102b40 0x7fbb4019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:06.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.803+0000 7fbb3ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb40102b40 0x7fbb4019d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57736/0 (socket says 172.21.15.114:57736) 2024-09-20T19:42:06.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.803+0000 7fbb3ffff640 1 -- 172.21.15.114:0/3609189580 learned_addr learned my addr 172.21.15.114:0/3609189580 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:06.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.803+0000 7fbb3f7fe640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb40103460 0x7fbb4019db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:06.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.803+0000 7fbb3ffff640 1 -- 172.21.15.114:0/3609189580 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb40103460 msgr2=0x7fbb4019db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:06.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.803+0000 7fbb3ffff640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb40103460 0x7fbb4019db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:06.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.803+0000 7fbb3ffff640 1 -- 172.21.15.114:0/3609189580 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb401977c0 con 0x7fbb40102b40 2024-09-20T19:42:06.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.803+0000 7fbb3ffff640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb40102b40 0x7fbb4019d640 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000d8d0 tx=0x7fbb3000dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:06.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.803+0000 7fbb3d7fa640 1 -- 172.21.15.114:0/3609189580 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fbb30017070 con 0x7fbb40102b40 2024-09-20T19:42:06.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.803+0000 7fbb46989640 1 -- 172.21.15.114:0/3609189580 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb40197ab0 con 0x7fbb40102b40 2024-09-20T19:42:06.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.804+0000 7fbb46989640 1 -- 172.21.15.114:0/3609189580 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb40198010 con 0x7fbb40102b40 2024-09-20T19:42:06.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.804+0000 7fbb3d7fa640 1 -- 172.21.15.114:0/3609189580 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbb30010460 con 0x7fbb40102b40 2024-09-20T19:42:06.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.804+0000 7fbb3d7fa640 1 -- 172.21.15.114:0/3609189580 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fbb3000f5f0 con 0x7fbb40102b40 2024-09-20T19:42:06.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.804+0000 7fbb46989640 1 -- 172.21.15.114:0/3609189580 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbb4006a370 con 0x7fbb40102b40 2024-09-20T19:42:06.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.805+0000 7fbb3d7fa640 1 -- 172.21.15.114:0/3609189580 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fbb30009a30 con 0x7fbb40102b40 2024-09-20T19:42:06.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.806+0000 7fbb3d7fa640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbb2806e310 0x7fbb280707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:06.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.806+0000 7fbb3f7fe640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbb2806e310 0x7fbb280707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:06.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.806+0000 7fbb3d7fa640 1 -- 172.21.15.114:0/3609189580 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7fbb3008dca0 con 0x7fbb40102b40 2024-09-20T19:42:06.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.806+0000 7fbb3d7fa640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493] conn(0x7fbb28074100 0x7fbb28076520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:06.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.806+0000 7fbb3d7fa640 1 -- 172.21.15.114:0/3609189580 --> [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb30004470 con 0x7fbb28074100 2024-09-20T19:42:06.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.806+0000 7fbb3d7fa640 1 -- 172.21.15.114:0/3609189580 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fbb30090310 con 0x7fbb40102b40 2024-09-20T19:42:06.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.806+0000 7fbb44f26640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493] conn(0x7fbb28074100 0x7fbb28076520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:06.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.806+0000 7fbb3f7fe640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbb2806e310 0x7fbb280707d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbb40198b10 tx=0x7fbb34004f30 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:06.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.807+0000 7fbb44f26640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493] conn(0x7fbb28074100 0x7fbb28076520 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:06.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.808+0000 7fbb3d7fa640 1 -- 172.21.15.114:0/3609189580 <== osd.2 v2:172.21.15.114:6818/4082826493 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbb30004470 con 0x7fbb28074100 2024-09-20T19:42:06.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.822+0000 7fbb46989640 1 -- 172.21.15.114:0/3609189580 --> [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbb40062880 con 0x7fbb28074100 2024-09-20T19:42:06.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.822+0000 7fbb3d7fa640 1 -- 172.21.15.114:0/3609189580 <== osd.2 v2:172.21.15.114:6818/4082826493 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbb40062880 con 0x7fbb28074100 2024-09-20T19:42:06.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.823+0000 7fbb46989640 1 -- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493] conn(0x7fbb28074100 msgr2=0x7fbb28076520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:06.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.823+0000 7fbb46989640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493] conn(0x7fbb28074100 0x7fbb28076520 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:06.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.823+0000 7fbb46989640 1 -- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbb2806e310 msgr2=0x7fbb280707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:06.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.823+0000 7fbb46989640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbb2806e310 0x7fbb280707d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbb40198b10 tx=0x7fbb34004f30 comp rx=0 tx=0).stop 2024-09-20T19:42:06.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.823+0000 7fbb46989640 1 -- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb40102b40 msgr2=0x7fbb4019d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:06.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.823+0000 7fbb46989640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb40102b40 0x7fbb4019d640 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000d8d0 tx=0x7fbb3000dd90 comp rx=0 tx=0).stop 2024-09-20T19:42:06.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.823+0000 7fbb46989640 1 -- 172.21.15.114:0/3609189580 shutdown_connections 2024-09-20T19:42:06.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.823+0000 7fbb46989640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:6818/4082826493,v1:172.21.15.114:6819/4082826493] conn(0x7fbb28074100 0x7fbb28076520 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:06.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.823+0000 7fbb46989640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fbb2806e310 0x7fbb280707d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:06.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.823+0000 7fbb46989640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fbb40103460 0x7fbb4019db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:06.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.823+0000 7fbb46989640 1 --2- 172.21.15.114:0/3609189580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb40102b40 0x7fbb4019d640 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:06.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.823+0000 7fbb46989640 1 -- 172.21.15.114:0/3609189580 >> 172.21.15.114:0/3609189580 conn(0x7fbb400fc4c0 msgr2=0x7fbb40103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:06.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.823+0000 7fbb46989640 1 -- 172.21.15.114:0/3609189580 shutdown_connections 2024-09-20T19:42:06.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.823+0000 7fbb46989640 1 -- 172.21.15.114:0/3609189580 wait complete. 2024-09-20T19:42:07.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.218+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/1235578079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84103420 msgr2=0x7f0c8410b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:07.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.218+0000 7f0c8ab9f640 1 --2- 172.21.15.114:0/1235578079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84103420 0x7f0c8410b910 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0c700098b0 tx=0x7f0c7002a560 comp rx=0 tx=0).stop 2024-09-20T19:42:07.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.218+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/1235578079 shutdown_connections 2024-09-20T19:42:07.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.218+0000 7f0c8ab9f640 1 --2- 172.21.15.114:0/1235578079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84103420 0x7f0c8410b910 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.218+0000 7f0c8ab9f640 1 --2- 172.21.15.114:0/1235578079 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84102b00 0x7f0c84102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.218+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/1235578079 >> 172.21.15.114:0/1235578079 conn(0x7f0c840fc4c0 msgr2=0x7f0c840fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:07.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.218+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/1235578079 shutdown_connections 2024-09-20T19:42:07.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.218+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/1235578079 wait complete. 2024-09-20T19:42:07.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.219+0000 7f0c8ab9f640 1 Processor -- start 2024-09-20T19:42:07.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.219+0000 7f0c8ab9f640 1 -- start start 2024-09-20T19:42:07.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.219+0000 7f0c8ab9f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84102b00 0x7f0c8419d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:07.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.219+0000 7f0c8ab9f640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84103420 0x7f0c8419dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:07.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.219+0000 7f0c8ab9f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8410edd0 con 0x7f0c84102b00 2024-09-20T19:42:07.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.219+0000 7f0c8ab9f640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8410ec50 con 0x7f0c84103420 2024-09-20T19:42:07.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.219+0000 7f0c8893b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84102b00 0x7f0c8419d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:07.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.219+0000 7f0c8893b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84102b00 0x7f0c8419d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57746/0 (socket says 172.21.15.114:57746) 2024-09-20T19:42:07.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.219+0000 7f0c8893b640 1 -- 172.21.15.114:0/2508812118 learned_addr learned my addr 172.21.15.114:0/2508812118 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:07.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.219+0000 7f0c83fff640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84103420 0x7f0c8419dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:07.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.219+0000 7f0c8893b640 1 -- 172.21.15.114:0/2508812118 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84103420 msgr2=0x7f0c8419dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:07.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.219+0000 7f0c8893b640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84103420 0x7f0c8419dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.219+0000 7f0c8893b640 1 -- 172.21.15.114:0/2508812118 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c841978a0 con 0x7f0c84102b00 2024-09-20T19:42:07.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.219+0000 7f0c8893b640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84102b00 0x7f0c8419d6e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0c7800d6f0 tx=0x7f0c7800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:07.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.219+0000 7f0c81ffb640 1 -- 172.21.15.114:0/2508812118 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0c78017070 con 0x7f0c84102b00 2024-09-20T19:42:07.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.220+0000 7f0c81ffb640 1 -- 172.21.15.114:0/2508812118 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0c7800de00 con 0x7f0c84102b00 2024-09-20T19:42:07.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.220+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/2508812118 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c84197b90 con 0x7f0c84102b00 2024-09-20T19:42:07.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.220+0000 7f0c81ffb640 1 -- 172.21.15.114:0/2508812118 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0c78004bb0 con 0x7f0c84102b00 2024-09-20T19:42:07.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.220+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/2508812118 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c84198120 con 0x7f0c84102b00 2024-09-20T19:42:07.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.220+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/2508812118 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0c84198660 con 0x7f0c84102b00 2024-09-20T19:42:07.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.221+0000 7f0c81ffb640 1 -- 172.21.15.114:0/2508812118 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f0c78004e60 con 0x7f0c84102b00 2024-09-20T19:42:07.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.221+0000 7f0c81ffb640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0c7406e2b0 0x7f0c74070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:07.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.221+0000 7f0c81ffb640 1 -- 172.21.15.114:0/2508812118 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f0c7808da20 con 0x7f0c84102b00 2024-09-20T19:42:07.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.222+0000 7f0c83fff640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0c7406e2b0 0x7f0c74070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:07.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.222+0000 7f0c81ffb640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.160:6808/3464010154,v1:172.21.15.160:6809/3464010154] conn(0x7f0c740740a0 0x7f0c740764c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:07.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.222+0000 7f0c81ffb640 1 -- 172.21.15.114:0/2508812118 --> [v2:172.21.15.160:6808/3464010154,v1:172.21.15.160:6809/3464010154] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c74076b70 con 0x7f0c740740a0 2024-09-20T19:42:07.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.222+0000 7f0c81ffb640 1 -- 172.21.15.114:0/2508812118 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f0c780922a0 con 0x7f0c84102b00 2024-09-20T19:42:07.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.222+0000 7f0c83fff640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0c7406e2b0 0x7f0c74070770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0c70000c00 tx=0x7f0c70031420 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:07.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.222+0000 7f0c8913c640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.160:6808/3464010154,v1:172.21.15.160:6809/3464010154] conn(0x7f0c740740a0 0x7f0c740764c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:07.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.222+0000 7f0c8913c640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.160:6808/3464010154,v1:172.21.15.160:6809/3464010154] conn(0x7f0c740740a0 0x7f0c740764c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:07.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.223+0000 7f0c81ffb640 1 -- 172.21.15.114:0/2508812118 <== osd.4 v2:172.21.15.160:6808/3464010154 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0c74076b70 con 0x7f0c740740a0 2024-09-20T19:42:07.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.236+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/2508812118 --> [v2:172.21.15.160:6808/3464010154,v1:172.21.15.160:6809/3464010154] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0c84108c30 con 0x7f0c740740a0 2024-09-20T19:42:07.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.237+0000 7f0c81ffb640 1 -- 172.21.15.114:0/2508812118 <== osd.4 v2:172.21.15.160:6808/3464010154 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0c84108c30 con 0x7f0c740740a0 2024-09-20T19:42:07.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.237+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/2508812118 >> [v2:172.21.15.160:6808/3464010154,v1:172.21.15.160:6809/3464010154] conn(0x7f0c740740a0 msgr2=0x7f0c740764c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:07.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.237+0000 7f0c8ab9f640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.160:6808/3464010154,v1:172.21.15.160:6809/3464010154] conn(0x7f0c740740a0 0x7f0c740764c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.237+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/2508812118 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0c7406e2b0 msgr2=0x7f0c74070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:07.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.237+0000 7f0c8ab9f640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0c7406e2b0 0x7f0c74070770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0c70000c00 tx=0x7f0c70031420 comp rx=0 tx=0).stop 2024-09-20T19:42:07.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.237+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/2508812118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84102b00 msgr2=0x7f0c8419d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:07.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.237+0000 7f0c8ab9f640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84102b00 0x7f0c8419d6e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0c7800d6f0 tx=0x7f0c7800dbb0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.237+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/2508812118 shutdown_connections 2024-09-20T19:42:07.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.237+0000 7f0c8ab9f640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.160:6808/3464010154,v1:172.21.15.160:6809/3464010154] conn(0x7f0c740740a0 0x7f0c740764c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.237+0000 7f0c8ab9f640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0c7406e2b0 0x7f0c74070770 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.237+0000 7f0c8ab9f640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c84103420 0x7f0c8419dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.237+0000 7f0c8ab9f640 1 --2- 172.21.15.114:0/2508812118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84102b00 0x7f0c8419d6e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.237+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/2508812118 >> 172.21.15.114:0/2508812118 conn(0x7f0c840fc4c0 msgr2=0x7f0c84103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:07.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.237+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/2508812118 shutdown_connections 2024-09-20T19:42:07.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:07.238+0000 7f0c8ab9f640 1 -- 172.21.15.114:0/2508812118 wait complete. 2024-09-20T19:42:07.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:07 smithi114 ceph-mon[28260]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:07.581 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:07 smithi160 ceph-mon[35859]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:07.870 INFO:teuthology.orchestra.run.smithi114.stdout:150323855372 2024-09-20T19:42:07.871 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-20T19:42:07.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.121+0000 7fba0a072640 1 -- 172.21.15.114:0/2538532560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba04101910 msgr2=0x7fba04101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:07.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.121+0000 7fba0a072640 1 --2- 172.21.15.114:0/2538532560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba04101910 0x7fba04101d70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec009880 tx=0x7fb9ec02a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.121+0000 7fba0a072640 1 -- 172.21.15.114:0/2538532560 shutdown_connections 2024-09-20T19:42:07.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.121+0000 7fba0a072640 1 --2- 172.21.15.114:0/2538532560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba04101910 0x7fba04101d70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.121+0000 7fba0a072640 1 --2- 172.21.15.114:0/2538532560 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba04107910 0x7fba04107cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.121+0000 7fba0a072640 1 -- 172.21.15.114:0/2538532560 >> 172.21.15.114:0/2538532560 conn(0x7fba040fd630 msgr2=0x7fba040ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.121+0000 7fba0a072640 1 -- 172.21.15.114:0/2538532560 shutdown_connections 2024-09-20T19:42:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.121+0000 7fba0a072640 1 -- 172.21.15.114:0/2538532560 wait complete. 2024-09-20T19:42:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.121+0000 7fba0a072640 1 Processor -- start 2024-09-20T19:42:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.121+0000 7fba0a072640 1 -- start start 2024-09-20T19:42:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.121+0000 7fba0a072640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba04101910 0x7fba0419d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.121+0000 7fba0a072640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba04107910 0x7fba0419db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.121+0000 7fba0a072640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fba0410edc0 con 0x7fba04101910 2024-09-20T19:42:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.121+0000 7fba0a072640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fba0410ec40 con 0x7fba04107910 2024-09-20T19:42:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.122+0000 7fba037fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba04101910 0x7fba0419d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:07.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.122+0000 7fba037fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba04101910 0x7fba0419d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57714/0 (socket says 172.21.15.114:57714) 2024-09-20T19:42:07.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.122+0000 7fba037fe640 1 -- 172.21.15.114:0/1176318332 learned_addr learned my addr 172.21.15.114:0/1176318332 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:07.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.122+0000 7fb9fbfff640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba04107910 0x7fba0419db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:07.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.122+0000 7fba037fe640 1 -- 172.21.15.114:0/1176318332 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba04107910 msgr2=0x7fba0419db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:07.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.122+0000 7fba037fe640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba04107910 0x7fba0419db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.122+0000 7fba037fe640 1 -- 172.21.15.114:0/1176318332 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba041977d0 con 0x7fba04101910 2024-09-20T19:42:07.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.122+0000 7fba037fe640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba04101910 0x7fba0419d650 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400c8a0 tx=0x7fb9f400cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:07.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.122+0000 7fba017fa640 1 -- 172.21.15.114:0/1176318332 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb9f40079b0 con 0x7fba04101910 2024-09-20T19:42:07.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.122+0000 7fba0a072640 1 -- 172.21.15.114:0/1176318332 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba04197ac0 con 0x7fba04101910 2024-09-20T19:42:07.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.122+0000 7fba017fa640 1 -- 172.21.15.114:0/1176318332 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb9f4007b50 con 0x7fba04101910 2024-09-20T19:42:07.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.122+0000 7fba017fa640 1 -- 172.21.15.114:0/1176318332 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb9f4018700 con 0x7fba04101910 2024-09-20T19:42:07.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.122+0000 7fba0a072640 1 -- 172.21.15.114:0/1176318332 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba04198050 con 0x7fba04101910 2024-09-20T19:42:07.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.123+0000 7fba0a072640 1 -- 172.21.15.114:0/1176318332 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fba04103050 con 0x7fba04101910 2024-09-20T19:42:07.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.124+0000 7fba017fa640 1 -- 172.21.15.114:0/1176318332 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fb9f4007d00 con 0x7fba04101910 2024-09-20T19:42:07.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.124+0000 7fba017fa640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fb9f006e2b0 0x7fb9f0070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:07.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.124+0000 7fb9fbfff640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fb9f006e2b0 0x7fb9f0070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:07.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.124+0000 7fba017fa640 1 -- 172.21.15.114:0/1176318332 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7fb9f408e0a0 con 0x7fba04101910 2024-09-20T19:42:07.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.124+0000 7fba017fa640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.160:6800/496816402,v1:172.21.15.160:6801/496816402] conn(0x7fb9f00740a0 0x7fb9f00764c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:07.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.124+0000 7fba017fa640 1 -- 172.21.15.114:0/1176318332 --> [v2:172.21.15.160:6800/496816402,v1:172.21.15.160:6801/496816402] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9f0076b70 con 0x7fb9f00740a0 2024-09-20T19:42:07.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.124+0000 7fba017fa640 1 -- 172.21.15.114:0/1176318332 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fb9f4095050 con 0x7fba04101910 2024-09-20T19:42:07.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.124+0000 7fba03fff640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.160:6800/496816402,v1:172.21.15.160:6801/496816402] conn(0x7fb9f00740a0 0x7fb9f00764c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:07.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.125+0000 7fb9fbfff640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fb9f006e2b0 0x7fb9f0070770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec031460 tx=0x7fb9ec009560 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:07.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.125+0000 7fba03fff640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.160:6800/496816402,v1:172.21.15.160:6801/496816402] conn(0x7fb9f00740a0 0x7fb9f00764c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:07.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.126+0000 7fba017fa640 1 -- 172.21.15.114:0/1176318332 <== osd.3 v2:172.21.15.160:6800/496816402 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9f0076b70 con 0x7fb9f00740a0 2024-09-20T19:42:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.139+0000 7fba0a072640 1 -- 172.21.15.114:0/1176318332 --> [v2:172.21.15.160:6800/496816402,v1:172.21.15.160:6801/496816402] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fba04062900 con 0x7fb9f00740a0 2024-09-20T19:42:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.139+0000 7fba017fa640 1 -- 172.21.15.114:0/1176318332 <== osd.3 v2:172.21.15.160:6800/496816402 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fba04062900 con 0x7fb9f00740a0 2024-09-20T19:42:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.140+0000 7fba0a072640 1 -- 172.21.15.114:0/1176318332 >> [v2:172.21.15.160:6800/496816402,v1:172.21.15.160:6801/496816402] conn(0x7fb9f00740a0 msgr2=0x7fb9f00764c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.140+0000 7fba0a072640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.160:6800/496816402,v1:172.21.15.160:6801/496816402] conn(0x7fb9f00740a0 0x7fb9f00764c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.140+0000 7fba0a072640 1 -- 172.21.15.114:0/1176318332 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fb9f006e2b0 msgr2=0x7fb9f0070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.140+0000 7fba0a072640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fb9f006e2b0 0x7fb9f0070770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec031460 tx=0x7fb9ec009560 comp rx=0 tx=0).stop 2024-09-20T19:42:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.140+0000 7fba0a072640 1 -- 172.21.15.114:0/1176318332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba04101910 msgr2=0x7fba0419d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.140+0000 7fba0a072640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba04101910 0x7fba0419d650 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400c8a0 tx=0x7fb9f400cd60 comp rx=0 tx=0).stop 2024-09-20T19:42:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.140+0000 7fba0a072640 1 -- 172.21.15.114:0/1176318332 shutdown_connections 2024-09-20T19:42:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.140+0000 7fba0a072640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fb9f006e2b0 0x7fb9f0070770 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.140+0000 7fba0a072640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.160:6800/496816402,v1:172.21.15.160:6801/496816402] conn(0x7fb9f00740a0 0x7fb9f00764c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.140+0000 7fba0a072640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fba04107910 0x7fba0419db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.140+0000 7fba0a072640 1 --2- 172.21.15.114:0/1176318332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba04101910 0x7fba0419d650 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:07.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.140+0000 7fba0a072640 1 -- 172.21.15.114:0/1176318332 >> 172.21.15.114:0/1176318332 conn(0x7fba040fd630 msgr2=0x7fba040fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:07.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.140+0000 7fba0a072640 1 -- 172.21.15.114:0/1176318332 shutdown_connections 2024-09-20T19:42:07.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:06.140+0000 7fba0a072640 1 -- 172.21.15.114:0/1176318332 wait complete. 2024-09-20T19:42:08.576 INFO:teuthology.orchestra.run.smithi114.stdout:115964117009 2024-09-20T19:42:08.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-20T19:42:09.038 INFO:teuthology.orchestra.run.smithi114.stdout:81604378646 2024-09-20T19:42:09.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-20T19:42:09.089 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:09.520 INFO:teuthology.orchestra.run.smithi114.stdout:171798691846 2024-09-20T19:42:09.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-20T19:42:09.559 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:09.645 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:09 smithi114 ceph-mon[28260]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:09.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:09 smithi160 ceph-mon[35859]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:09.799 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:10.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.640+0000 7f5cd18f0640 1 -- 172.21.15.114:0/598874878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ccc1067e0 msgr2=0x7f5ccc106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:10.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.640+0000 7f5cd18f0640 1 --2- 172.21.15.114:0/598874878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ccc1067e0 0x7f5ccc106bc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc009880 tx=0x7f5cbc02a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.640+0000 7f5cd18f0640 1 -- 172.21.15.114:0/598874878 shutdown_connections 2024-09-20T19:42:10.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.640+0000 7f5cd18f0640 1 --2- 172.21.15.114:0/598874878 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ccc1007e0 0x7f5ccc100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.640+0000 7f5cd18f0640 1 --2- 172.21.15.114:0/598874878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ccc1067e0 0x7f5ccc106bc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.640+0000 7f5cd18f0640 1 -- 172.21.15.114:0/598874878 >> 172.21.15.114:0/598874878 conn(0x7f5ccc0fc4c0 msgr2=0x7f5ccc0fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:10.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.640+0000 7f5cd18f0640 1 -- 172.21.15.114:0/598874878 shutdown_connections 2024-09-20T19:42:10.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.640+0000 7f5cd18f0640 1 -- 172.21.15.114:0/598874878 wait complete. 2024-09-20T19:42:10.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.641+0000 7f5cd18f0640 1 Processor -- start 2024-09-20T19:42:10.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.641+0000 7f5cd18f0640 1 -- start start 2024-09-20T19:42:10.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.641+0000 7f5cd18f0640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ccc1007e0 0x7f5ccc19d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:10.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.641+0000 7f5cd18f0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ccc1067e0 0x7f5ccc19db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:10.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.641+0000 7f5cd18f0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc10dcb0 con 0x7f5ccc1067e0 2024-09-20T19:42:10.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.641+0000 7f5cd18f0640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc10db30 con 0x7f5ccc1007e0 2024-09-20T19:42:10.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.641+0000 7f5cca7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ccc1067e0 0x7f5ccc19db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:10.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.641+0000 7f5cca7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ccc1067e0 0x7f5ccc19db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57794/0 (socket says 172.21.15.114:57794) 2024-09-20T19:42:10.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.641+0000 7f5cca7fc640 1 -- 172.21.15.114:0/2888909512 learned_addr learned my addr 172.21.15.114:0/2888909512 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:10.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.642+0000 7f5cca7fc640 1 -- 172.21.15.114:0/2888909512 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ccc1007e0 msgr2=0x7f5ccc19d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:10.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.642+0000 7f5cca7fc640 1 --2- 172.21.15.114:0/2888909512 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ccc1007e0 0x7f5ccc19d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.642+0000 7f5cca7fc640 1 -- 172.21.15.114:0/2888909512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ccc197790 con 0x7f5ccc1067e0 2024-09-20T19:42:10.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.642+0000 7f5cca7fc640 1 --2- 172.21.15.114:0/2888909512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ccc1067e0 0x7f5ccc19db50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000d6f0 tx=0x7f5cc000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:10.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.642+0000 7f5cd08ee640 1 -- 172.21.15.114:0/2888909512 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5cc0017070 con 0x7f5ccc1067e0 2024-09-20T19:42:10.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.642+0000 7f5cd08ee640 1 -- 172.21.15.114:0/2888909512 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5cc000de00 con 0x7f5ccc1067e0 2024-09-20T19:42:10.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.642+0000 7f5cd18f0640 1 -- 172.21.15.114:0/2888909512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ccc197a80 con 0x7f5ccc1067e0 2024-09-20T19:42:10.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.642+0000 7f5cd08ee640 1 -- 172.21.15.114:0/2888909512 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5cc0004bb0 con 0x7f5ccc1067e0 2024-09-20T19:42:10.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.642+0000 7f5cd18f0640 1 -- 172.21.15.114:0/2888909512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ccc198010 con 0x7f5ccc1067e0 2024-09-20T19:42:10.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.643+0000 7f5cd18f0640 1 -- 172.21.15.114:0/2888909512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ccc101f20 con 0x7f5ccc1067e0 2024-09-20T19:42:10.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.644+0000 7f5cd08ee640 1 -- 172.21.15.114:0/2888909512 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f5cc0004e60 con 0x7f5ccc1067e0 2024-09-20T19:42:10.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.644+0000 7f5cd08ee640 1 --2- 172.21.15.114:0/2888909512 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5cb406e2b0 0x7f5cb4070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:10.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.644+0000 7f5cd08ee640 1 -- 172.21.15.114:0/2888909512 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f5cc008da20 con 0x7f5ccc1067e0 2024-09-20T19:42:10.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.644+0000 7f5ccaffd640 1 --2- 172.21.15.114:0/2888909512 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5cb406e2b0 0x7f5cb4070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:10.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.645+0000 7f5ccaffd640 1 --2- 172.21.15.114:0/2888909512 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5cb406e2b0 0x7f5cb4070770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0096c0 tx=0x7f5cbc000bc0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:10.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.649+0000 7f5cd08ee640 1 -- 172.21.15.114:0/2888909512 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f5cc005b6c0 con 0x7f5ccc1067e0 2024-09-20T19:42:10.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.836+0000 7f5cd18f0640 1 -- 172.21.15.114:0/2888909512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f5ccc109d60 con 0x7f5ccc1067e0 2024-09-20T19:42:10.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.836+0000 7f5cd08ee640 1 -- 172.21.15.114:0/2888909512 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5cc005f370 con 0x7f5ccc1067e0 2024-09-20T19:42:10.839 INFO:teuthology.orchestra.run.smithi114.stdout:38654705699 2024-09-20T19:42:10.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.839+0000 7f5cd18f0640 1 -- 172.21.15.114:0/2888909512 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5cb406e2b0 msgr2=0x7f5cb4070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:10.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.839+0000 7f5cd18f0640 1 --2- 172.21.15.114:0/2888909512 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5cb406e2b0 0x7f5cb4070770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0096c0 tx=0x7f5cbc000bc0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.839+0000 7f5cd18f0640 1 -- 172.21.15.114:0/2888909512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ccc1067e0 msgr2=0x7f5ccc19db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:10.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.839+0000 7f5cd18f0640 1 --2- 172.21.15.114:0/2888909512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ccc1067e0 0x7f5ccc19db50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000d6f0 tx=0x7f5cc000dbb0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.839+0000 7f5cd18f0640 1 -- 172.21.15.114:0/2888909512 shutdown_connections 2024-09-20T19:42:10.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.839+0000 7f5cd18f0640 1 --2- 172.21.15.114:0/2888909512 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5cb406e2b0 0x7f5cb4070770 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.839+0000 7f5cd18f0640 1 --2- 172.21.15.114:0/2888909512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ccc1067e0 0x7f5ccc19db50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.839+0000 7f5cd18f0640 1 --2- 172.21.15.114:0/2888909512 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5ccc1007e0 0x7f5ccc19d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.839+0000 7f5cd18f0640 1 -- 172.21.15.114:0/2888909512 >> 172.21.15.114:0/2888909512 conn(0x7f5ccc0fc4c0 msgr2=0x7f5ccc069e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:10.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.839+0000 7f5cd18f0640 1 -- 172.21.15.114:0/2888909512 shutdown_connections 2024-09-20T19:42:10.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.839+0000 7f5cd18f0640 1 -- 172.21.15.114:0/2888909512 wait complete. 2024-09-20T19:42:10.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.925+0000 7fc74281e640 1 -- 172.21.15.114:0/1855280912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc73c103460 msgr2=0x7fc73c10b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.925+0000 7fc74281e640 1 --2- 172.21.15.114:0/1855280912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc73c103460 0x7fc73c10b950 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc7300098b0 tx=0x7fc73002a560 comp rx=0 tx=0).stop 2024-09-20T19:42:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.925+0000 7fc74281e640 1 -- 172.21.15.114:0/1855280912 shutdown_connections 2024-09-20T19:42:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.925+0000 7fc74281e640 1 --2- 172.21.15.114:0/1855280912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc73c103460 0x7fc73c10b950 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.925+0000 7fc74281e640 1 --2- 172.21.15.114:0/1855280912 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc73c102b40 0x7fc73c102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.925+0000 7fc74281e640 1 -- 172.21.15.114:0/1855280912 >> 172.21.15.114:0/1855280912 conn(0x7fc73c0fc520 msgr2=0x7fc73c0fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.925+0000 7fc74281e640 1 -- 172.21.15.114:0/1855280912 shutdown_connections 2024-09-20T19:42:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.925+0000 7fc74281e640 1 -- 172.21.15.114:0/1855280912 wait complete. 2024-09-20T19:42:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.926+0000 7fc74281e640 1 Processor -- start 2024-09-20T19:42:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.926+0000 7fc74281e640 1 -- start start 2024-09-20T19:42:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.926+0000 7fc74281e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc73c102b40 0x7fc73c19d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.926+0000 7fc74281e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc73c103460 0x7fc73c19dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:10.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.926+0000 7fc74281e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc73c10ee70 con 0x7fc73c103460 2024-09-20T19:42:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.926+0000 7fc74281e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc73c10ecf0 con 0x7fc73c102b40 2024-09-20T19:42:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.926+0000 7fc73b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc73c103460 0x7fc73c19dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.926+0000 7fc73b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc73c103460 0x7fc73c19dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57806/0 (socket says 172.21.15.114:57806) 2024-09-20T19:42:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.926+0000 7fc73b7fe640 1 -- 172.21.15.114:0/1638195977 learned_addr learned my addr 172.21.15.114:0/1638195977 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.926+0000 7fc73bfff640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc73c102b40 0x7fc73c19d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.927+0000 7fc73b7fe640 1 -- 172.21.15.114:0/1638195977 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc73c102b40 msgr2=0x7fc73c19d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.927+0000 7fc73b7fe640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc73c102b40 0x7fc73c19d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.927+0000 7fc73b7fe640 1 -- 172.21.15.114:0/1638195977 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73c197860 con 0x7fc73c103460 2024-09-20T19:42:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.927+0000 7fc73bfff640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc73c102b40 0x7fc73c19d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-20T19:42:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.927+0000 7fc73b7fe640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc73c103460 0x7fc73c19dbe0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc730031460 tx=0x7fc7300048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.927+0000 7fc7397fa640 1 -- 172.21.15.114:0/1638195977 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc73003f070 con 0x7fc73c103460 2024-09-20T19:42:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.927+0000 7fc7397fa640 1 -- 172.21.15.114:0/1638195977 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc7300044d0 con 0x7fc73c103460 2024-09-20T19:42:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.927+0000 7fc7397fa640 1 -- 172.21.15.114:0/1638195977 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc730035030 con 0x7fc73c103460 2024-09-20T19:42:10.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.927+0000 7fc74281e640 1 -- 172.21.15.114:0/1638195977 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc73c197af0 con 0x7fc73c103460 2024-09-20T19:42:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.927+0000 7fc74281e640 1 -- 172.21.15.114:0/1638195977 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc73c198020 con 0x7fc73c103460 2024-09-20T19:42:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.928+0000 7fc74281e640 1 -- 172.21.15.114:0/1638195977 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc73c1985e0 con 0x7fc73c103460 2024-09-20T19:42:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.928+0000 7fc7397fa640 1 -- 172.21.15.114:0/1638195977 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fc730042070 con 0x7fc73c103460 2024-09-20T19:42:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.929+0000 7fc7397fa640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc72406e310 0x7fc7240707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.929+0000 7fc7397fa640 1 -- 172.21.15.114:0/1638195977 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7fc7300ae7f0 con 0x7fc73c103460 2024-09-20T19:42:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.929+0000 7fc73bfff640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc72406e310 0x7fc7240707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.929+0000 7fc7397fa640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805] conn(0x7fc724074100 0x7fc724076520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.929+0000 7fc7397fa640 1 -- 172.21.15.114:0/1638195977 --> [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc7300054e0 con 0x7fc724074100 2024-09-20T19:42:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.929+0000 7fc7397fa640 1 -- 172.21.15.114:0/1638195977 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fc7300b32a0 con 0x7fc73c103460 2024-09-20T19:42:10.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.929+0000 7fc740dbb640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805] conn(0x7fc724074100 0x7fc724076520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.929+0000 7fc73bfff640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc72406e310 0x7fc7240707d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc72c006fd0 tx=0x7fc72c008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.930+0000 7fc740dbb640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805] conn(0x7fc724074100 0x7fc724076520 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.930+0000 7fc7397fa640 1 -- 172.21.15.114:0/1638195977 <== osd.1 v2:172.21.15.114:6810/326057805 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc7300054e0 con 0x7fc724074100 2024-09-20T19:42:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.943+0000 7fc74281e640 1 -- 172.21.15.114:0/1638195977 --> [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc73c062900 con 0x7fc724074100 2024-09-20T19:42:10.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.944+0000 7fc7397fa640 1 -- 172.21.15.114:0/1638195977 <== osd.1 v2:172.21.15.114:6810/326057805 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc73c062900 con 0x7fc724074100 2024-09-20T19:42:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.944+0000 7fc74281e640 1 -- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805] conn(0x7fc724074100 msgr2=0x7fc724076520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.944+0000 7fc74281e640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805] conn(0x7fc724074100 0x7fc724076520 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.944+0000 7fc74281e640 1 -- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc72406e310 msgr2=0x7fc7240707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.944+0000 7fc74281e640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc72406e310 0x7fc7240707d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc72c006fd0 tx=0x7fc72c008040 comp rx=0 tx=0).stop 2024-09-20T19:42:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.944+0000 7fc74281e640 1 -- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc73c103460 msgr2=0x7fc73c19dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.944+0000 7fc74281e640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc73c103460 0x7fc73c19dbe0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc730031460 tx=0x7fc7300048f0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.944+0000 7fc74281e640 1 -- 172.21.15.114:0/1638195977 shutdown_connections 2024-09-20T19:42:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.944+0000 7fc74281e640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc72406e310 0x7fc7240707d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.944+0000 7fc74281e640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:6810/326057805,v1:172.21.15.114:6811/326057805] conn(0x7fc724074100 0x7fc724076520 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.944+0000 7fc74281e640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc73c103460 0x7fc73c19dbe0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.944+0000 7fc74281e640 1 --2- 172.21.15.114:0/1638195977 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc73c102b40 0x7fc73c19d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.944+0000 7fc74281e640 1 -- 172.21.15.114:0/1638195977 >> 172.21.15.114:0/1638195977 conn(0x7fc73c0fc520 msgr2=0x7fc73c103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:10.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.944+0000 7fc74281e640 1 -- 172.21.15.114:0/1638195977 shutdown_connections 2024-09-20T19:42:10.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.944+0000 7fc74281e640 1 -- 172.21.15.114:0/1638195977 wait complete. 2024-09-20T19:42:11.608 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:11 smithi114 ceph-mon[28260]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:11.609 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:11 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1418770281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T19:42:11.609 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:11 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/2888909512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T19:42:11.609 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:11 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:42:11.638 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705698 got 38654705699 for osd.0 2024-09-20T19:42:11.638 DEBUG:teuthology.parallel:result is None 2024-09-20T19:42:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.522+0000 7f155e539640 1 -- 172.21.15.114:0/2010862105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15581067e0 msgr2=0x7f1558106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.522+0000 7f155e539640 1 --2- 172.21.15.114:0/2010862105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15581067e0 0x7f1558106bc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1548009880 tx=0x7f154802a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.522+0000 7f155e539640 1 -- 172.21.15.114:0/2010862105 shutdown_connections 2024-09-20T19:42:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.522+0000 7f155e539640 1 --2- 172.21.15.114:0/2010862105 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f15581007e0 0x7f1558100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.522+0000 7f155e539640 1 --2- 172.21.15.114:0/2010862105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15581067e0 0x7f1558106bc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.522+0000 7f155e539640 1 -- 172.21.15.114:0/2010862105 >> 172.21.15.114:0/2010862105 conn(0x7f15580fc4c0 msgr2=0x7f15580fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.522+0000 7f155e539640 1 -- 172.21.15.114:0/2010862105 shutdown_connections 2024-09-20T19:42:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.522+0000 7f155e539640 1 -- 172.21.15.114:0/2010862105 wait complete. 2024-09-20T19:42:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.522+0000 7f155e539640 1 Processor -- start 2024-09-20T19:42:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.523+0000 7f155e539640 1 -- start start 2024-09-20T19:42:11.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.523+0000 7f155e539640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15581007e0 0x7f155819d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.523+0000 7f155e539640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f15581067e0 0x7f155819dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.523+0000 7f155e539640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f155810dd10 con 0x7f15581007e0 2024-09-20T19:42:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.523+0000 7f155e539640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f155810db90 con 0x7f15581067e0 2024-09-20T19:42:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.523+0000 7f1557fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15581007e0 0x7f155819d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.523+0000 7f1557fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15581007e0 0x7f155819d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57772/0 (socket says 172.21.15.114:57772) 2024-09-20T19:42:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.523+0000 7f1557fff640 1 -- 172.21.15.114:0/1418770281 learned_addr learned my addr 172.21.15.114:0/1418770281 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.523+0000 7f1557fff640 1 -- 172.21.15.114:0/1418770281 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f15581067e0 msgr2=0x7f155819dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.523+0000 7f1557fff640 1 --2- 172.21.15.114:0/1418770281 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f15581067e0 0x7f155819dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.523+0000 7f1557fff640 1 -- 172.21.15.114:0/1418770281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1558197800 con 0x7f15581007e0 2024-09-20T19:42:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.523+0000 7f1557fff640 1 --2- 172.21.15.114:0/1418770281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15581007e0 0x7f155819d680 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1548031460 tx=0x7f15480048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.524+0000 7f15557fa640 1 -- 172.21.15.114:0/1418770281 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f154803f070 con 0x7f15581007e0 2024-09-20T19:42:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.524+0000 7f15557fa640 1 -- 172.21.15.114:0/1418770281 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1548004cf0 con 0x7f15581007e0 2024-09-20T19:42:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.524+0000 7f155e539640 1 -- 172.21.15.114:0/1418770281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1558197a90 con 0x7f15581007e0 2024-09-20T19:42:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.524+0000 7f15557fa640 1 -- 172.21.15.114:0/1418770281 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f1548035030 con 0x7f15581007e0 2024-09-20T19:42:11.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.524+0000 7f155e539640 1 -- 172.21.15.114:0/1418770281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1558197fc0 con 0x7f15581007e0 2024-09-20T19:42:11.761 INFO:teuthology.orchestra.run.smithi114.stdout:150323855372 2024-09-20T19:42:11.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.525+0000 7f155e539640 1 -- 172.21.15.114:0/1418770281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1558101f20 con 0x7f15581007e0 2024-09-20T19:42:11.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.525+0000 7f15557fa640 1 -- 172.21.15.114:0/1418770281 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f1548042070 con 0x7f15581007e0 2024-09-20T19:42:11.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.526+0000 7f15557fa640 1 --2- 172.21.15.114:0/1418770281 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f154006e2b0 0x7f1540070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:11.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.526+0000 7f15557fa640 1 -- 172.21.15.114:0/1418770281 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f15480ae910 con 0x7f15581007e0 2024-09-20T19:42:11.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.526+0000 7f15577fe640 1 --2- 172.21.15.114:0/1418770281 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f154006e2b0 0x7f1540070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:11.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.526+0000 7f15577fe640 1 --2- 172.21.15.114:0/1418770281 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f154006e2b0 0x7f1540070770 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1558105f00 tx=0x7f154c006c90 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:11.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.530+0000 7f15557fa640 1 -- 172.21.15.114:0/1418770281 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f154807c5b0 con 0x7f15581007e0 2024-09-20T19:42:11.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.706+0000 7f155e539640 1 -- 172.21.15.114:0/1418770281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f1558109d60 con 0x7f15581007e0 2024-09-20T19:42:11.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.706+0000 7f15557fa640 1 -- 172.21.15.114:0/1418770281 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1548080260 con 0x7f15581007e0 2024-09-20T19:42:11.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.709+0000 7f155e539640 1 -- 172.21.15.114:0/1418770281 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f154006e2b0 msgr2=0x7f1540070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:11.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.709+0000 7f155e539640 1 --2- 172.21.15.114:0/1418770281 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f154006e2b0 0x7f1540070770 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1558105f00 tx=0x7f154c006c90 comp rx=0 tx=0).stop 2024-09-20T19:42:11.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.709+0000 7f155e539640 1 -- 172.21.15.114:0/1418770281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15581007e0 msgr2=0x7f155819d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:11.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.709+0000 7f155e539640 1 --2- 172.21.15.114:0/1418770281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15581007e0 0x7f155819d680 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1548031460 tx=0x7f15480048f0 comp rx=0 tx=0).stop 2024-09-20T19:42:11.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.709+0000 7f155e539640 1 -- 172.21.15.114:0/1418770281 shutdown_connections 2024-09-20T19:42:11.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.709+0000 7f155e539640 1 --2- 172.21.15.114:0/1418770281 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f154006e2b0 0x7f1540070770 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:11.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.709+0000 7f155e539640 1 --2- 172.21.15.114:0/1418770281 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f15581067e0 0x7f155819dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:11.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.709+0000 7f155e539640 1 --2- 172.21.15.114:0/1418770281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15581007e0 0x7f155819d680 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:11.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.709+0000 7f155e539640 1 -- 172.21.15.114:0/1418770281 >> 172.21.15.114:0/1418770281 conn(0x7f15580fc4c0 msgr2=0x7f1558069e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:11.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.709+0000 7f155e539640 1 -- 172.21.15.114:0/1418770281 shutdown_connections 2024-09-20T19:42:11.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:10.709+0000 7f155e539640 1 -- 172.21.15.114:0/1418770281 wait complete. 2024-09-20T19:42:11.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:11 smithi160 ceph-mon[35859]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:11.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:11 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1418770281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T19:42:11.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:11 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/2888909512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T19:42:11.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:11 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:42:12.554 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855372 got 150323855372 for osd.4 2024-09-20T19:42:12.554 DEBUG:teuthology.parallel:result is None 2024-09-20T19:42:13.025 INFO:teuthology.orchestra.run.smithi114.stdout:60129542173 2024-09-20T19:42:13.025 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-20T19:42:13.052 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:13.080 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:13.091 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:13.595 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:13 smithi114 ceph-mon[28260]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:13.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:13 smithi160 ceph-mon[35859]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:13.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.909+0000 7f4a67d9c640 1 -- 172.21.15.114:0/3214036294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a60102ae0 msgr2=0x7f4a60102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:13.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.909+0000 7f4a67d9c640 1 --2- 172.21.15.114:0/3214036294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a60102ae0 0x7f4a60102ec0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009880 tx=0x7f4a5002a530 comp rx=0 tx=0).stop 2024-09-20T19:42:13.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.909+0000 7f4a67d9c640 1 -- 172.21.15.114:0/3214036294 shutdown_connections 2024-09-20T19:42:13.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.909+0000 7f4a67d9c640 1 --2- 172.21.15.114:0/3214036294 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4a60103400 0x7f4a6010b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:13.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.909+0000 7f4a67d9c640 1 --2- 172.21.15.114:0/3214036294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a60102ae0 0x7f4a60102ec0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:13.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.909+0000 7f4a67d9c640 1 -- 172.21.15.114:0/3214036294 >> 172.21.15.114:0/3214036294 conn(0x7f4a600fc520 msgr2=0x7f4a600fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.909+0000 7f4a67d9c640 1 -- 172.21.15.114:0/3214036294 shutdown_connections 2024-09-20T19:42:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.909+0000 7f4a67d9c640 1 -- 172.21.15.114:0/3214036294 wait complete. 2024-09-20T19:42:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.910+0000 7f4a67d9c640 1 Processor -- start 2024-09-20T19:42:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.910+0000 7f4a67d9c640 1 -- start start 2024-09-20T19:42:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.910+0000 7f4a67d9c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a60102ae0 0x7f4a6019d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.910+0000 7f4a67d9c640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4a60103400 0x7f4a6019dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.910+0000 7f4a67d9c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6010ee70 con 0x7f4a60102ae0 2024-09-20T19:42:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.910+0000 7f4a67d9c640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6010ecf0 con 0x7f4a60103400 2024-09-20T19:42:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.910+0000 7f4a65b38640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a60102ae0 0x7f4a6019d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.910+0000 7f4a65b38640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a60102ae0 0x7f4a6019d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57834/0 (socket says 172.21.15.114:57834) 2024-09-20T19:42:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.910+0000 7f4a65b38640 1 -- 172.21.15.114:0/2973653652 learned_addr learned my addr 172.21.15.114:0/2973653652 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.910+0000 7f4a65b38640 1 -- 172.21.15.114:0/2973653652 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4a60103400 msgr2=0x7f4a6019dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:13.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.910+0000 7f4a65b38640 1 --2- 172.21.15.114:0/2973653652 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4a60103400 0x7f4a6019dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.910+0000 7f4a65b38640 1 -- 172.21.15.114:0/2973653652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a60197860 con 0x7f4a60102ae0 2024-09-20T19:42:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.911+0000 7f4a65b38640 1 --2- 172.21.15.114:0/2973653652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a60102ae0 0x7f4a6019d6a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4a50031460 tx=0x7f4a500048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.911+0000 7f4a56ffd640 1 -- 172.21.15.114:0/2973653652 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f4a5003f070 con 0x7f4a60102ae0 2024-09-20T19:42:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.911+0000 7f4a56ffd640 1 -- 172.21.15.114:0/2973653652 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4a50004cf0 con 0x7f4a60102ae0 2024-09-20T19:42:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.911+0000 7f4a67d9c640 1 -- 172.21.15.114:0/2973653652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a60197af0 con 0x7f4a60102ae0 2024-09-20T19:42:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.911+0000 7f4a56ffd640 1 -- 172.21.15.114:0/2973653652 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f4a50035030 con 0x7f4a60102ae0 2024-09-20T19:42:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.911+0000 7f4a67d9c640 1 -- 172.21.15.114:0/2973653652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a60198020 con 0x7f4a60102ae0 2024-09-20T19:42:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.912+0000 7f4a67d9c640 1 -- 172.21.15.114:0/2973653652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a6006abe0 con 0x7f4a60102ae0 2024-09-20T19:42:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.912+0000 7f4a56ffd640 1 -- 172.21.15.114:0/2973653652 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f4a50042070 con 0x7f4a60102ae0 2024-09-20T19:42:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.913+0000 7f4a56ffd640 1 --2- 172.21.15.114:0/2973653652 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f4a4c06e310 0x7f4a4c0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.913+0000 7f4a56ffd640 1 -- 172.21.15.114:0/2973653652 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f4a500ae840 con 0x7f4a60102ae0 2024-09-20T19:42:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.913+0000 7f4a65337640 1 --2- 172.21.15.114:0/2973653652 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f4a4c06e310 0x7f4a4c0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:13.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.913+0000 7f4a65337640 1 --2- 172.21.15.114:0/2973653652 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f4a4c06e310 0x7f4a4c0707d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4a600fe510 tx=0x7f4a5c006d30 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:13.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:13.917+0000 7f4a56ffd640 1 -- 172.21.15.114:0/2973653652 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f4a5007c560 con 0x7f4a60102ae0 2024-09-20T19:42:14.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.088+0000 7f4a67d9c640 1 -- 172.21.15.114:0/2973653652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f4a60198d90 con 0x7f4a60102ae0 2024-09-20T19:42:14.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.088+0000 7f4a56ffd640 1 -- 172.21.15.114:0/2973653652 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4a50080210 con 0x7f4a60102ae0 2024-09-20T19:42:14.090 INFO:teuthology.orchestra.run.smithi114.stdout:115964117010 2024-09-20T19:42:14.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.091+0000 7f4a67d9c640 1 -- 172.21.15.114:0/2973653652 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f4a4c06e310 msgr2=0x7f4a4c0707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:14.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.091+0000 7f4a67d9c640 1 --2- 172.21.15.114:0/2973653652 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f4a4c06e310 0x7f4a4c0707d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4a600fe510 tx=0x7f4a5c006d30 comp rx=0 tx=0).stop 2024-09-20T19:42:14.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.091+0000 7f4a67d9c640 1 -- 172.21.15.114:0/2973653652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a60102ae0 msgr2=0x7f4a6019d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:14.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.091+0000 7f4a67d9c640 1 --2- 172.21.15.114:0/2973653652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a60102ae0 0x7f4a6019d6a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4a50031460 tx=0x7f4a500048f0 comp rx=0 tx=0).stop 2024-09-20T19:42:14.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.091+0000 7f4a67d9c640 1 -- 172.21.15.114:0/2973653652 shutdown_connections 2024-09-20T19:42:14.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.091+0000 7f4a67d9c640 1 --2- 172.21.15.114:0/2973653652 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f4a4c06e310 0x7f4a4c0707d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:14.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.091+0000 7f4a67d9c640 1 --2- 172.21.15.114:0/2973653652 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4a60103400 0x7f4a6019dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:14.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.091+0000 7f4a67d9c640 1 --2- 172.21.15.114:0/2973653652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a60102ae0 0x7f4a6019d6a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:14.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.091+0000 7f4a67d9c640 1 -- 172.21.15.114:0/2973653652 >> 172.21.15.114:0/2973653652 conn(0x7f4a600fc520 msgr2=0x7f4a60108eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:14.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.091+0000 7f4a67d9c640 1 -- 172.21.15.114:0/2973653652 shutdown_connections 2024-09-20T19:42:14.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.091+0000 7f4a67d9c640 1 -- 172.21.15.114:0/2973653652 wait complete. 2024-09-20T19:42:14.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.129+0000 7f006460a640 1 -- 172.21.15.114:0/160633047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f005c1007f0 msgr2=0x7f005c100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:14.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.129+0000 7f006460a640 1 --2- 172.21.15.114:0/160633047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f005c1007f0 0x7f005c100c50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f004c009880 tx=0x7f004c02a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:14.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.129+0000 7f006460a640 1 -- 172.21.15.114:0/160633047 shutdown_connections 2024-09-20T19:42:14.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.129+0000 7f006460a640 1 --2- 172.21.15.114:0/160633047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f005c1007f0 0x7f005c100c50 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:14.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.129+0000 7f006460a640 1 --2- 172.21.15.114:0/160633047 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f005c1067f0 0x7f005c106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:14.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.129+0000 7f006460a640 1 -- 172.21.15.114:0/160633047 >> 172.21.15.114:0/160633047 conn(0x7f005c0fc4f0 msgr2=0x7f005c0fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:14.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.129+0000 7f006460a640 1 -- 172.21.15.114:0/160633047 shutdown_connections 2024-09-20T19:42:14.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.129+0000 7f006460a640 1 -- 172.21.15.114:0/160633047 wait complete. 2024-09-20T19:42:14.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.129+0000 7f006460a640 1 Processor -- start 2024-09-20T19:42:14.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.129+0000 7f006460a640 1 -- start start 2024-09-20T19:42:14.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f006460a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f005c1007f0 0x7f005c19d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:14.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f006460a640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f005c1067f0 0x7f005c19dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:14.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f006460a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f005c10ee30 con 0x7f005c1007f0 2024-09-20T19:42:14.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f006460a640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f005c10ecb0 con 0x7f005c1067f0 2024-09-20T19:42:14.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f00623a6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f005c1007f0 0x7f005c19d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:14.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f00623a6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f005c1007f0 0x7f005c19d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57850/0 (socket says 172.21.15.114:57850) 2024-09-20T19:42:14.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f00623a6640 1 -- 172.21.15.114:0/930197623 learned_addr learned my addr 172.21.15.114:0/930197623 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:14.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f0061ba5640 1 --2- 172.21.15.114:0/930197623 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f005c1067f0 0x7f005c19dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:14.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f00623a6640 1 -- 172.21.15.114:0/930197623 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f005c1067f0 msgr2=0x7f005c19dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:14.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f00623a6640 1 --2- 172.21.15.114:0/930197623 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f005c1067f0 0x7f005c19dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:14.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f00623a6640 1 -- 172.21.15.114:0/930197623 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005c1977e0 con 0x7f005c1007f0 2024-09-20T19:42:14.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f00623a6640 1 --2- 172.21.15.114:0/930197623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f005c1007f0 0x7f005c19d660 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f005800b420 tx=0x7f005800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:14.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f00537fe640 1 -- 172.21.15.114:0/930197623 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0058011630 con 0x7f005c1007f0 2024-09-20T19:42:14.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f006460a640 1 -- 172.21.15.114:0/930197623 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f005c197ad0 con 0x7f005c1007f0 2024-09-20T19:42:14.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f00537fe640 1 -- 172.21.15.114:0/930197623 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0058016070 con 0x7f005c1007f0 2024-09-20T19:42:14.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f00537fe640 1 -- 172.21.15.114:0/930197623 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0058010d40 con 0x7f005c1007f0 2024-09-20T19:42:14.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.130+0000 7f006460a640 1 -- 172.21.15.114:0/930197623 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f005c198030 con 0x7f005c1007f0 2024-09-20T19:42:14.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.132+0000 7f006460a640 1 -- 172.21.15.114:0/930197623 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f005c101f30 con 0x7f005c1007f0 2024-09-20T19:42:14.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.132+0000 7f00537fe640 1 -- 172.21.15.114:0/930197623 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f005801b460 con 0x7f005c1007f0 2024-09-20T19:42:14.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.132+0000 7f00537fe640 1 --2- 172.21.15.114:0/930197623 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f004806e2b0 0x7f0048070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:14.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.132+0000 7f00537fe640 1 -- 172.21.15.114:0/930197623 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f005808d680 con 0x7f005c1007f0 2024-09-20T19:42:14.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.132+0000 7f0061ba5640 1 --2- 172.21.15.114:0/930197623 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f004806e2b0 0x7f0048070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:14.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.133+0000 7f0061ba5640 1 --2- 172.21.15.114:0/930197623 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f004806e2b0 0x7f0048070770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f005c198b30 tx=0x7f004c031400 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:14.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.137+0000 7f00537fe640 1 -- 172.21.15.114:0/930197623 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f005805b320 con 0x7f005c1007f0 2024-09-20T19:42:14.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.312+0000 7f006460a640 1 -- 172.21.15.114:0/930197623 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f005c108d80 con 0x7f005c1007f0 2024-09-20T19:42:14.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.312+0000 7f00537fe640 1 -- 172.21.15.114:0/930197623 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f005805efd0 con 0x7f005c1007f0 2024-09-20T19:42:14.314 INFO:teuthology.orchestra.run.smithi114.stdout:171798691847 2024-09-20T19:42:14.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.314+0000 7f006460a640 1 -- 172.21.15.114:0/930197623 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f004806e2b0 msgr2=0x7f0048070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.314+0000 7f006460a640 1 --2- 172.21.15.114:0/930197623 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f004806e2b0 0x7f0048070770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f005c198b30 tx=0x7f004c031400 comp rx=0 tx=0).stop 2024-09-20T19:42:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.314+0000 7f006460a640 1 -- 172.21.15.114:0/930197623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f005c1007f0 msgr2=0x7f005c19d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.314+0000 7f006460a640 1 --2- 172.21.15.114:0/930197623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f005c1007f0 0x7f005c19d660 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f005800b420 tx=0x7f005800b8e0 comp rx=0 tx=0).stop 2024-09-20T19:42:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.314+0000 7f006460a640 1 -- 172.21.15.114:0/930197623 shutdown_connections 2024-09-20T19:42:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.314+0000 7f006460a640 1 --2- 172.21.15.114:0/930197623 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f004806e2b0 0x7f0048070770 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.314+0000 7f006460a640 1 --2- 172.21.15.114:0/930197623 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f005c1067f0 0x7f005c19dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.314+0000 7f006460a640 1 --2- 172.21.15.114:0/930197623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f005c1007f0 0x7f005c19d660 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.314+0000 7f006460a640 1 -- 172.21.15.114:0/930197623 >> 172.21.15.114:0/930197623 conn(0x7f005c0fc4f0 msgr2=0x7f005c0fcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.314+0000 7f006460a640 1 -- 172.21.15.114:0/930197623 shutdown_connections 2024-09-20T19:42:14.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:14.314+0000 7f006460a640 1 -- 172.21.15.114:0/930197623 wait complete. 2024-09-20T19:42:14.627 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:14 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/2973653652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T19:42:14.628 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:14 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/930197623' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T19:42:14.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:14 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/2973653652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T19:42:14.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:14 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/930197623' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T19:42:15.033 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117009 got 115964117010 for osd.3 2024-09-20T19:42:15.034 DEBUG:teuthology.parallel:result is None 2024-09-20T19:42:15.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.232+0000 7ff5edde8640 1 -- 172.21.15.114:0/4172633855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e81007e0 msgr2=0x7ff5e8100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:15.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.232+0000 7ff5edde8640 1 --2- 172.21.15.114:0/4172633855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e81007e0 0x7ff5e8100c40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8009880 tx=0x7ff5d802a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:15.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.232+0000 7ff5edde8640 1 -- 172.21.15.114:0/4172633855 shutdown_connections 2024-09-20T19:42:15.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.232+0000 7ff5edde8640 1 --2- 172.21.15.114:0/4172633855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e81007e0 0x7ff5e8100c40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:15.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.232+0000 7ff5edde8640 1 --2- 172.21.15.114:0/4172633855 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff5e81067e0 0x7ff5e8106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:15.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.232+0000 7ff5edde8640 1 -- 172.21.15.114:0/4172633855 >> 172.21.15.114:0/4172633855 conn(0x7ff5e80fc4c0 msgr2=0x7ff5e80fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:15.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.232+0000 7ff5edde8640 1 -- 172.21.15.114:0/4172633855 shutdown_connections 2024-09-20T19:42:15.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.233+0000 7ff5edde8640 1 -- 172.21.15.114:0/4172633855 wait complete. 2024-09-20T19:42:15.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.233+0000 7ff5edde8640 1 Processor -- start 2024-09-20T19:42:15.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.233+0000 7ff5edde8640 1 -- start start 2024-09-20T19:42:15.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5edde8640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff5e81007e0 0x7ff5e81987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:15.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5edde8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e81067e0 0x7ff5e8198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:15.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5edde8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e810dcb0 con 0x7ff5e81067e0 2024-09-20T19:42:15.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5edde8640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e810db30 con 0x7ff5e81007e0 2024-09-20T19:42:15.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5e6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e81067e0 0x7ff5e8198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:15.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5e6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e81067e0 0x7ff5e8198d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57870/0 (socket says 172.21.15.114:57870) 2024-09-20T19:42:15.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5e6ffd640 1 -- 172.21.15.114:0/1137674413 learned_addr learned my addr 172.21.15.114:0/1137674413 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:15.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5e77fe640 1 --2- 172.21.15.114:0/1137674413 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff5e81007e0 0x7ff5e81987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:15.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5e6ffd640 1 -- 172.21.15.114:0/1137674413 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff5e81007e0 msgr2=0x7ff5e81987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:15.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5e6ffd640 1 --2- 172.21.15.114:0/1137674413 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff5e81007e0 0x7ff5e81987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:15.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5e6ffd640 1 -- 172.21.15.114:0/1137674413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e8199320 con 0x7ff5e81067e0 2024-09-20T19:42:15.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5e6ffd640 1 --2- 172.21.15.114:0/1137674413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e81067e0 0x7ff5e8198d20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8031460 tx=0x7ff5d80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:15.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5e4ff9640 1 -- 172.21.15.114:0/1137674413 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff5d803f070 con 0x7ff5e81067e0 2024-09-20T19:42:15.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5e4ff9640 1 -- 172.21.15.114:0/1137674413 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff5d80035f0 con 0x7ff5e81067e0 2024-09-20T19:42:15.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5edde8640 1 -- 172.21.15.114:0/1137674413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e81995b0 con 0x7ff5e81067e0 2024-09-20T19:42:15.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.234+0000 7ff5e4ff9640 1 -- 172.21.15.114:0/1137674413 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff5d8035030 con 0x7ff5e81067e0 2024-09-20T19:42:15.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.235+0000 7ff5edde8640 1 -- 172.21.15.114:0/1137674413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e81a1b60 con 0x7ff5e81067e0 2024-09-20T19:42:15.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.235+0000 7ff5edde8640 1 -- 172.21.15.114:0/1137674413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e8101f20 con 0x7ff5e81067e0 2024-09-20T19:42:15.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.236+0000 7ff5e4ff9640 1 -- 172.21.15.114:0/1137674413 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7ff5d8042070 con 0x7ff5e81067e0 2024-09-20T19:42:15.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.237+0000 7ff5e4ff9640 1 --2- 172.21.15.114:0/1137674413 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff5d406e2b0 0x7ff5d4070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:15.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.237+0000 7ff5e4ff9640 1 -- 172.21.15.114:0/1137674413 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7ff5d80ae740 con 0x7ff5e81067e0 2024-09-20T19:42:15.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.237+0000 7ff5e77fe640 1 --2- 172.21.15.114:0/1137674413 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff5d406e2b0 0x7ff5d4070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:15.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.237+0000 7ff5e77fe640 1 --2- 172.21.15.114:0/1137674413 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff5d406e2b0 0x7ff5d4070770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0005fd0 tx=0x7ff5d0009680 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:15.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.241+0000 7ff5e4ff9640 1 -- 172.21.15.114:0/1137674413 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7ff5d8080090 con 0x7ff5e81067e0 2024-09-20T19:42:15.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.414+0000 7ff5edde8640 1 -- 172.21.15.114:0/1137674413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7ff5e8108d50 con 0x7ff5e81067e0 2024-09-20T19:42:15.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.415+0000 7ff5e4ff9640 1 -- 172.21.15.114:0/1137674413 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff5d80848c0 con 0x7ff5e81067e0 2024-09-20T19:42:15.416 INFO:teuthology.orchestra.run.smithi114.stdout:81604378647 2024-09-20T19:42:15.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.417+0000 7ff5edde8640 1 -- 172.21.15.114:0/1137674413 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff5d406e2b0 msgr2=0x7ff5d4070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:15.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.417+0000 7ff5edde8640 1 --2- 172.21.15.114:0/1137674413 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff5d406e2b0 0x7ff5d4070770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0005fd0 tx=0x7ff5d0009680 comp rx=0 tx=0).stop 2024-09-20T19:42:15.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.417+0000 7ff5edde8640 1 -- 172.21.15.114:0/1137674413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e81067e0 msgr2=0x7ff5e8198d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:15.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.417+0000 7ff5edde8640 1 --2- 172.21.15.114:0/1137674413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e81067e0 0x7ff5e8198d20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8031460 tx=0x7ff5d80045e0 comp rx=0 tx=0).stop 2024-09-20T19:42:15.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.417+0000 7ff5edde8640 1 -- 172.21.15.114:0/1137674413 shutdown_connections 2024-09-20T19:42:15.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.417+0000 7ff5edde8640 1 --2- 172.21.15.114:0/1137674413 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff5d406e2b0 0x7ff5d4070770 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:15.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.417+0000 7ff5edde8640 1 --2- 172.21.15.114:0/1137674413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e81067e0 0x7ff5e8198d20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:15.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.417+0000 7ff5edde8640 1 --2- 172.21.15.114:0/1137674413 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff5e81007e0 0x7ff5e81987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:15.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.417+0000 7ff5edde8640 1 -- 172.21.15.114:0/1137674413 >> 172.21.15.114:0/1137674413 conn(0x7ff5e80fc4c0 msgr2=0x7ff5e80fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:15.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.417+0000 7ff5edde8640 1 -- 172.21.15.114:0/1137674413 shutdown_connections 2024-09-20T19:42:15.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:15.417+0000 7ff5edde8640 1 -- 172.21.15.114:0/1137674413 wait complete. 2024-09-20T19:42:15.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:15 smithi114 ceph-mon[28260]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:15.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:15 smithi160 ceph-mon[35859]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:15.787 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691846 got 171798691847 for osd.5 2024-09-20T19:42:15.787 DEBUG:teuthology.parallel:result is None 2024-09-20T19:42:16.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:16 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1137674413' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T19:42:16.514 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:16.531 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378646 got 81604378647 for osd.2 2024-09-20T19:42:16.531 DEBUG:teuthology.parallel:result is None 2024-09-20T19:42:16.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:16 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1137674413' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T19:42:16.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.927+0000 7efd265a4640 1 -- 172.21.15.114:0/3839070139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd20107930 msgr2=0x7efd20107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:16.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.927+0000 7efd265a4640 1 --2- 172.21.15.114:0/3839070139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd20107930 0x7efd20107d10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efd10009880 tx=0x7efd1002a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:16.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.927+0000 7efd265a4640 1 -- 172.21.15.114:0/3839070139 shutdown_connections 2024-09-20T19:42:16.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.927+0000 7efd265a4640 1 --2- 172.21.15.114:0/3839070139 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efd20101930 0x7efd20101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:16.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.927+0000 7efd265a4640 1 --2- 172.21.15.114:0/3839070139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd20107930 0x7efd20107d10 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:16.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.927+0000 7efd265a4640 1 -- 172.21.15.114:0/3839070139 >> 172.21.15.114:0/3839070139 conn(0x7efd200fd630 msgr2=0x7efd200ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:16.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.927+0000 7efd265a4640 1 -- 172.21.15.114:0/3839070139 shutdown_connections 2024-09-20T19:42:16.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.927+0000 7efd265a4640 1 -- 172.21.15.114:0/3839070139 wait complete. 2024-09-20T19:42:16.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.928+0000 7efd265a4640 1 Processor -- start 2024-09-20T19:42:16.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.928+0000 7efd265a4640 1 -- start start 2024-09-20T19:42:16.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.928+0000 7efd265a4640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efd20101930 0x7efd2019d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:16.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.928+0000 7efd265a4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd20107930 0x7efd2019dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:16.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.928+0000 7efd265a4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd2010ee00 con 0x7efd20107930 2024-09-20T19:42:16.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.928+0000 7efd265a4640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7efd2010ec80 con 0x7efd20101930 2024-09-20T19:42:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.928+0000 7efd1f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd20107930 0x7efd2019dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.928+0000 7efd1f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd20107930 0x7efd2019dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42682/0 (socket says 172.21.15.114:42682) 2024-09-20T19:42:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.928+0000 7efd1f7fe640 1 -- 172.21.15.114:0/614190434 learned_addr learned my addr 172.21.15.114:0/614190434 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.929+0000 7efd1ffff640 1 --2- 172.21.15.114:0/614190434 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efd20101930 0x7efd2019d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.929+0000 7efd1f7fe640 1 -- 172.21.15.114:0/614190434 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efd20101930 msgr2=0x7efd2019d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.929+0000 7efd1f7fe640 1 --2- 172.21.15.114:0/614190434 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efd20101930 0x7efd2019d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.929+0000 7efd1f7fe640 1 -- 172.21.15.114:0/614190434 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd201977f0 con 0x7efd20107930 2024-09-20T19:42:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.929+0000 7efd1f7fe640 1 --2- 172.21.15.114:0/614190434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd20107930 0x7efd2019dbb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efd0800d6f0 tx=0x7efd0800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.929+0000 7efd1d7fa640 1 -- 172.21.15.114:0/614190434 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7efd08017070 con 0x7efd20107930 2024-09-20T19:42:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.929+0000 7efd1d7fa640 1 -- 172.21.15.114:0/614190434 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efd0800de00 con 0x7efd20107930 2024-09-20T19:42:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.929+0000 7efd265a4640 1 -- 172.21.15.114:0/614190434 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd20197ae0 con 0x7efd20107930 2024-09-20T19:42:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.929+0000 7efd1d7fa640 1 -- 172.21.15.114:0/614190434 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7efd08004bb0 con 0x7efd20107930 2024-09-20T19:42:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.929+0000 7efd265a4640 1 -- 172.21.15.114:0/614190434 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd20198070 con 0x7efd20107930 2024-09-20T19:42:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.930+0000 7efd265a4640 1 -- 172.21.15.114:0/614190434 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd20103070 con 0x7efd20107930 2024-09-20T19:42:16.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.931+0000 7efd1d7fa640 1 -- 172.21.15.114:0/614190434 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7efd08004e60 con 0x7efd20107930 2024-09-20T19:42:16.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.931+0000 7efd1d7fa640 1 --2- 172.21.15.114:0/614190434 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7efd0c06e2b0 0x7efd0c070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:16.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.931+0000 7efd1ffff640 1 --2- 172.21.15.114:0/614190434 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7efd0c06e2b0 0x7efd0c070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:16.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.931+0000 7efd1d7fa640 1 -- 172.21.15.114:0/614190434 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7efd0808d970 con 0x7efd20107930 2024-09-20T19:42:16.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.932+0000 7efd1ffff640 1 --2- 172.21.15.114:0/614190434 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7efd0c06e2b0 0x7efd0c070770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efd100096c0 tx=0x7efd10031400 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:16.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:16.936+0000 7efd1d7fa640 1 -- 172.21.15.114:0/614190434 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7efd0808fe20 con 0x7efd20107930 2024-09-20T19:42:17.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:17.101+0000 7efd265a4640 1 -- 172.21.15.114:0/614190434 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7efd20109ea0 con 0x7efd20107930 2024-09-20T19:42:17.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:17.101+0000 7efd1d7fa640 1 -- 172.21.15.114:0/614190434 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efd0805f2c0 con 0x7efd20107930 2024-09-20T19:42:17.103 INFO:teuthology.orchestra.run.smithi114.stdout:60129542174 2024-09-20T19:42:17.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:17.104+0000 7efd265a4640 1 -- 172.21.15.114:0/614190434 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7efd0c06e2b0 msgr2=0x7efd0c070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:17.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:17.104+0000 7efd265a4640 1 --2- 172.21.15.114:0/614190434 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7efd0c06e2b0 0x7efd0c070770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efd100096c0 tx=0x7efd10031400 comp rx=0 tx=0).stop 2024-09-20T19:42:17.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:17.104+0000 7efd265a4640 1 -- 172.21.15.114:0/614190434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd20107930 msgr2=0x7efd2019dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:17.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:17.104+0000 7efd265a4640 1 --2- 172.21.15.114:0/614190434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd20107930 0x7efd2019dbb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efd0800d6f0 tx=0x7efd0800dbb0 comp rx=0 tx=0).stop 2024-09-20T19:42:17.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:17.104+0000 7efd265a4640 1 -- 172.21.15.114:0/614190434 shutdown_connections 2024-09-20T19:42:17.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:17.104+0000 7efd265a4640 1 --2- 172.21.15.114:0/614190434 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7efd0c06e2b0 0x7efd0c070770 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:17.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:17.104+0000 7efd265a4640 1 --2- 172.21.15.114:0/614190434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd20107930 0x7efd2019dbb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:17.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:17.104+0000 7efd265a4640 1 --2- 172.21.15.114:0/614190434 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7efd20101930 0x7efd2019d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:17.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:17.104+0000 7efd265a4640 1 -- 172.21.15.114:0/614190434 >> 172.21.15.114:0/614190434 conn(0x7efd200fd630 msgr2=0x7efd200fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:17.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:17.104+0000 7efd265a4640 1 -- 172.21.15.114:0/614190434 shutdown_connections 2024-09-20T19:42:17.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:17.104+0000 7efd265a4640 1 -- 172.21.15.114:0/614190434 wait complete. 2024-09-20T19:42:17.361 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:17 smithi114 ceph-mon[28260]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:17.361 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:17 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/614190434' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T19:42:17.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:17 smithi160 ceph-mon[35859]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:17.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:17 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/614190434' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T19:42:17.781 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542173 got 60129542174 for osd.1 2024-09-20T19:42:17.781 DEBUG:teuthology.parallel:result is None 2024-09-20T19:42:17.781 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-20T19:42:17.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:42:18.040 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:18.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.535+0000 7f89ecfb4640 1 -- 172.21.15.114:0/2738470794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89e81067e0 msgr2=0x7f89e8106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:18.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.535+0000 7f89ecfb4640 1 --2- 172.21.15.114:0/2738470794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89e81067e0 0x7f89e8106bc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f89e0009880 tx=0x7f89e002a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:18.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.535+0000 7f89ecfb4640 1 -- 172.21.15.114:0/2738470794 shutdown_connections 2024-09-20T19:42:18.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.535+0000 7f89ecfb4640 1 --2- 172.21.15.114:0/2738470794 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f89e81007e0 0x7f89e8100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:18.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.535+0000 7f89ecfb4640 1 --2- 172.21.15.114:0/2738470794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89e81067e0 0x7f89e8106bc0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:18.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.535+0000 7f89ecfb4640 1 -- 172.21.15.114:0/2738470794 >> 172.21.15.114:0/2738470794 conn(0x7f89e80fc4c0 msgr2=0x7f89e80fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:18.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.535+0000 7f89ecfb4640 1 -- 172.21.15.114:0/2738470794 shutdown_connections 2024-09-20T19:42:18.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.535+0000 7f89ecfb4640 1 -- 172.21.15.114:0/2738470794 wait complete. 2024-09-20T19:42:18.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.535+0000 7f89ecfb4640 1 Processor -- start 2024-09-20T19:42:18.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.535+0000 7f89ecfb4640 1 -- start start 2024-09-20T19:42:18.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89ecfb4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89e81007e0 0x7f89e819d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:18.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89ecfb4640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f89e81067e0 0x7f89e819dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:18.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89ecfb4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f89e810dd10 con 0x7f89e81007e0 2024-09-20T19:42:18.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89ecfb4640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f89e810db90 con 0x7f89e81067e0 2024-09-20T19:42:18.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89e659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89e81007e0 0x7f89e819d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:18.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89e659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89e81007e0 0x7f89e819d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42702/0 (socket says 172.21.15.114:42702) 2024-09-20T19:42:18.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89e659c640 1 -- 172.21.15.114:0/238043886 learned_addr learned my addr 172.21.15.114:0/238043886 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:18.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89e5d9b640 1 --2- 172.21.15.114:0/238043886 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f89e81067e0 0x7f89e819dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:18.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89e659c640 1 -- 172.21.15.114:0/238043886 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f89e81067e0 msgr2=0x7f89e819dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:18.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89e659c640 1 --2- 172.21.15.114:0/238043886 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f89e81067e0 0x7f89e819dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:18.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89e659c640 1 -- 172.21.15.114:0/238043886 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e81977f0 con 0x7f89e81007e0 2024-09-20T19:42:18.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89e5d9b640 1 --2- 172.21.15.114:0/238043886 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f89e81067e0 0x7f89e819dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-20T19:42:18.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89e659c640 1 --2- 172.21.15.114:0/238043886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89e81007e0 0x7f89e819d670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f89e0031460 tx=0x7f89e00048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:18.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89d77fe640 1 -- 172.21.15.114:0/238043886 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f89e003f070 con 0x7f89e81007e0 2024-09-20T19:42:18.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89d77fe640 1 -- 172.21.15.114:0/238043886 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f89e0004cf0 con 0x7f89e81007e0 2024-09-20T19:42:18.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89d77fe640 1 -- 172.21.15.114:0/238043886 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f89e0035030 con 0x7f89e81007e0 2024-09-20T19:42:18.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89ecfb4640 1 -- 172.21.15.114:0/238043886 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e8197a80 con 0x7f89e81007e0 2024-09-20T19:42:18.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.536+0000 7f89ecfb4640 1 -- 172.21.15.114:0/238043886 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e8197fb0 con 0x7f89e81007e0 2024-09-20T19:42:18.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.538+0000 7f89ecfb4640 1 -- 172.21.15.114:0/238043886 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89e8101f20 con 0x7f89e81007e0 2024-09-20T19:42:18.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.538+0000 7f89d77fe640 1 -- 172.21.15.114:0/238043886 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f89e0042070 con 0x7f89e81007e0 2024-09-20T19:42:18.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.538+0000 7f89d77fe640 1 --2- 172.21.15.114:0/238043886 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f89d006e310 0x7f89d00707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:18.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.539+0000 7f89d77fe640 1 -- 172.21.15.114:0/238043886 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f89e00ae840 con 0x7f89e81007e0 2024-09-20T19:42:18.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.539+0000 7f89e5d9b640 1 --2- 172.21.15.114:0/238043886 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f89d006e310 0x7f89d00707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:18.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.539+0000 7f89e5d9b640 1 --2- 172.21.15.114:0/238043886 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f89d006e310 0x7f89d00707d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f89e8105d30 tx=0x7f89dc0073c0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:18.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.543+0000 7f89d77fe640 1 -- 172.21.15.114:0/238043886 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f89e007c560 con 0x7f89e81007e0 2024-09-20T19:42:18.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.707+0000 7f89ecfb4640 1 -- 172.21.15.114:0/238043886 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f89e8198260 con 0x7f89d006e310 2024-09-20T19:42:18.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.708+0000 7f89d77fe640 1 -- 172.21.15.114:0/238043886 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+12321 (secure 0 0 0) 0x7f89e8198260 con 0x7f89d006e310 2024-09-20T19:42:18.710 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:42:18.710 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-20T19:42:18.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.711+0000 7f89ecfb4640 1 -- 172.21.15.114:0/238043886 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f89d006e310 msgr2=0x7f89d00707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:18.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.711+0000 7f89ecfb4640 1 --2- 172.21.15.114:0/238043886 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f89d006e310 0x7f89d00707d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f89e8105d30 tx=0x7f89dc0073c0 comp rx=0 tx=0).stop 2024-09-20T19:42:18.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.711+0000 7f89ecfb4640 1 -- 172.21.15.114:0/238043886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89e81007e0 msgr2=0x7f89e819d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:18.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.711+0000 7f89ecfb4640 1 --2- 172.21.15.114:0/238043886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89e81007e0 0x7f89e819d670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f89e0031460 tx=0x7f89e00048f0 comp rx=0 tx=0).stop 2024-09-20T19:42:18.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.711+0000 7f89ecfb4640 1 -- 172.21.15.114:0/238043886 shutdown_connections 2024-09-20T19:42:18.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.711+0000 7f89ecfb4640 1 --2- 172.21.15.114:0/238043886 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f89d006e310 0x7f89d00707d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:18.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.711+0000 7f89ecfb4640 1 --2- 172.21.15.114:0/238043886 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f89e81067e0 0x7f89e819dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:18.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.711+0000 7f89ecfb4640 1 --2- 172.21.15.114:0/238043886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89e81007e0 0x7f89e819d670 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:18.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.711+0000 7f89ecfb4640 1 -- 172.21.15.114:0/238043886 >> 172.21.15.114:0/238043886 conn(0x7f89e80fc4c0 msgr2=0x7f89e81045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:18.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.711+0000 7f89ecfb4640 1 -- 172.21.15.114:0/238043886 shutdown_connections 2024-09-20T19:42:18.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:18.711+0000 7f89ecfb4640 1 -- 172.21.15.114:0/238043886 wait complete. 2024-09-20T19:42:19.331 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-09-20T19:42:18.268243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":178592,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":562292320,"statfs":{"total":575970213888,"available":575787335680,"internally_reserved":0,"allocated":4018176,"data_stored":2628582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":178707105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001965"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":191,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-20T19:41:49.631452+0000","last_change":"2024-09-20T19:41:22.091442+0000","last_active":"2024-09-20T19:41:49.631452+0000","last_peered":"2024-09-20T19:41:49.631452+0000","last_clean":"2024-09-20T19:41:49.631452+0000","last_became_active":"2024-09-20T19:41:22.091021+0000","last_became_peered":"2024-09-20T19:41:22.091021+0000","last_unstale":"2024-09-20T19:41:49.631452+0000","last_undegraded":"2024-09-20T19:41:49.631452+0000","last_fullsized":"2024-09-20T19:41:49.631452+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:40:33.239148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:40:33.239148+0000","last_clean_scrub_stamp":"2024-09-20T19:40:33.239148+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:24:39.025082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691848,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31552,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713600,"statfs":{"total":95995035648,"available":95962726400,"internally_reserved":0,"allocated":372736,"data_stored":142913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27392,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717760,"statfs":{"total":95995035648,"available":95966986240,"internally_reserved":0,"allocated":372736,"data_stored":142913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117011,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":966656,"data_stored":733281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378648,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27392,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717760,"statfs":{"total":95995035648,"available":95966986240,"internally_reserved":0,"allocated":372736,"data_stored":142913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542175,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":966656,"data_stored":733281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705700,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":966656,"data_stored":733281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:42:19.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T19:42:19.579 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:19.699 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:19 smithi114 ceph-mon[28260]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:19.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:19 smithi160 ceph-mon[35859]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:20.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.023+0000 7f7b0485e640 1 -- 172.21.15.114:0/2731987891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b00106820 msgr2=0x7f7b00106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:20.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.023+0000 7f7b0485e640 1 --2- 172.21.15.114:0/2731987891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b00106820 0x7f7b00106c00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7af0009880 tx=0x7f7af002a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:20.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.023+0000 7f7b0485e640 1 -- 172.21.15.114:0/2731987891 shutdown_connections 2024-09-20T19:42:20.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.023+0000 7f7b0485e640 1 --2- 172.21.15.114:0/2731987891 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b00100820 0x7f7b00100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:20.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.023+0000 7f7b0485e640 1 --2- 172.21.15.114:0/2731987891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b00106820 0x7f7b00106c00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:20.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.023+0000 7f7b0485e640 1 -- 172.21.15.114:0/2731987891 >> 172.21.15.114:0/2731987891 conn(0x7f7b000fc520 msgr2=0x7f7b000fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:20.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.023+0000 7f7b0485e640 1 -- 172.21.15.114:0/2731987891 shutdown_connections 2024-09-20T19:42:20.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.023+0000 7f7b0485e640 1 -- 172.21.15.114:0/2731987891 wait complete. 2024-09-20T19:42:20.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.024+0000 7f7b0485e640 1 Processor -- start 2024-09-20T19:42:20.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.024+0000 7f7b0485e640 1 -- start start 2024-09-20T19:42:20.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.024+0000 7f7b0485e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b00100820 0x7f7b0019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.024+0000 7f7b0485e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b00106820 0x7f7b0019cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.024+0000 7f7b0485e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0010dd50 con 0x7f7b00106820 2024-09-20T19:42:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.024+0000 7f7b0485e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0010dbd0 con 0x7f7b00100820 2024-09-20T19:42:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.024+0000 7f7af7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b00106820 0x7f7b0019cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.024+0000 7f7af7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b00106820 0x7f7b0019cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42718/0 (socket says 172.21.15.114:42718) 2024-09-20T19:42:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.024+0000 7f7af7fff640 1 -- 172.21.15.114:0/4021865717 learned_addr learned my addr 172.21.15.114:0/4021865717 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.025+0000 7f7afe59c640 1 --2- 172.21.15.114:0/4021865717 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b00100820 0x7f7b0019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.025+0000 7f7af7fff640 1 -- 172.21.15.114:0/4021865717 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b00100820 msgr2=0x7f7b0019c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.025+0000 7f7af7fff640 1 --2- 172.21.15.114:0/4021865717 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b00100820 0x7f7b0019c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.025+0000 7f7af7fff640 1 -- 172.21.15.114:0/4021865717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b00196720 con 0x7f7b00106820 2024-09-20T19:42:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.025+0000 7f7afe59c640 1 --2- 172.21.15.114:0/4021865717 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b00100820 0x7f7b0019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-20T19:42:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.025+0000 7f7af7fff640 1 --2- 172.21.15.114:0/4021865717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b00106820 0x7f7b0019cb90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800b420 tx=0x7f7ae800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.025+0000 7f7af77fe640 1 -- 172.21.15.114:0/4021865717 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f7ae8011630 con 0x7f7b00106820 2024-09-20T19:42:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.025+0000 7f7af77fe640 1 -- 172.21.15.114:0/4021865717 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7ae8016070 con 0x7f7b00106820 2024-09-20T19:42:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.025+0000 7f7b0485e640 1 -- 172.21.15.114:0/4021865717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b00196a10 con 0x7f7b00106820 2024-09-20T19:42:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.025+0000 7f7af77fe640 1 -- 172.21.15.114:0/4021865717 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f7ae8010d40 con 0x7f7b00106820 2024-09-20T19:42:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.025+0000 7f7b0485e640 1 -- 172.21.15.114:0/4021865717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b00196f70 con 0x7f7b00106820 2024-09-20T19:42:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.026+0000 7f7b0485e640 1 -- 172.21.15.114:0/4021865717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b00101f60 con 0x7f7b00106820 2024-09-20T19:42:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.026+0000 7f7af77fe640 1 -- 172.21.15.114:0/4021865717 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f7ae80117d0 con 0x7f7b00106820 2024-09-20T19:42:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.027+0000 7f7af77fe640 1 --2- 172.21.15.114:0/4021865717 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7aec06e2b0 0x7f7aec070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.027+0000 7f7afe59c640 1 --2- 172.21.15.114:0/4021865717 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7aec06e2b0 0x7f7aec070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.027+0000 7f7af77fe640 1 -- 172.21.15.114:0/4021865717 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f7ae808d660 con 0x7f7b00106820 2024-09-20T19:42:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.028+0000 7f7afe59c640 1 --2- 172.21.15.114:0/4021865717 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7aec06e2b0 0x7f7aec070770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7af0031460 tx=0x7f7af0004280 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.031+0000 7f7af77fe640 1 -- 172.21.15.114:0/4021865717 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f7ae805b300 con 0x7f7b00106820 2024-09-20T19:42:20.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.204+0000 7f7b0485e640 1 -- 172.21.15.114:0/4021865717 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7b00197280 con 0x7f7aec06e2b0 2024-09-20T19:42:20.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.205+0000 7f7af77fe640 1 -- 172.21.15.114:0/4021865717 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+12321 (secure 0 0 0) 0x7f7b00197280 con 0x7f7aec06e2b0 2024-09-20T19:42:20.208 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:42:20.208 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-20T19:42:20.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.207+0000 7f7b0485e640 1 -- 172.21.15.114:0/4021865717 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7aec06e2b0 msgr2=0x7f7aec070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:20.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.207+0000 7f7b0485e640 1 --2- 172.21.15.114:0/4021865717 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7aec06e2b0 0x7f7aec070770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7af0031460 tx=0x7f7af0004280 comp rx=0 tx=0).stop 2024-09-20T19:42:20.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.207+0000 7f7b0485e640 1 -- 172.21.15.114:0/4021865717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b00106820 msgr2=0x7f7b0019cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:20.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.207+0000 7f7b0485e640 1 --2- 172.21.15.114:0/4021865717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b00106820 0x7f7b0019cb90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800b420 tx=0x7f7ae800b8e0 comp rx=0 tx=0).stop 2024-09-20T19:42:20.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.207+0000 7f7b0485e640 1 -- 172.21.15.114:0/4021865717 shutdown_connections 2024-09-20T19:42:20.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.207+0000 7f7b0485e640 1 --2- 172.21.15.114:0/4021865717 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7aec06e2b0 0x7f7aec070770 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:20.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.207+0000 7f7b0485e640 1 --2- 172.21.15.114:0/4021865717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b00106820 0x7f7b0019cb90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:20.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.207+0000 7f7b0485e640 1 --2- 172.21.15.114:0/4021865717 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b00100820 0x7f7b0019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:20.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.207+0000 7f7b0485e640 1 -- 172.21.15.114:0/4021865717 >> 172.21.15.114:0/4021865717 conn(0x7f7b000fc520 msgr2=0x7f7b000fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:20.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.207+0000 7f7b0485e640 1 -- 172.21.15.114:0/4021865717 shutdown_connections 2024-09-20T19:42:20.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:20.207+0000 7f7b0485e640 1 -- 172.21.15.114:0/4021865717 wait complete. 2024-09-20T19:42:20.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:20 smithi114 ceph-mon[28260]: from='client.14506 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:42:20.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:20 smithi160 ceph-mon[35859]: from='client.14506 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:42:20.826 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-09-20T19:42:18.268243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":178592,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":562292320,"statfs":{"total":575970213888,"available":575787335680,"internally_reserved":0,"allocated":4018176,"data_stored":2628582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":178707105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001965"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":191,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-20T19:41:49.631452+0000","last_change":"2024-09-20T19:41:22.091442+0000","last_active":"2024-09-20T19:41:49.631452+0000","last_peered":"2024-09-20T19:41:49.631452+0000","last_clean":"2024-09-20T19:41:49.631452+0000","last_became_active":"2024-09-20T19:41:22.091021+0000","last_became_peered":"2024-09-20T19:41:22.091021+0000","last_unstale":"2024-09-20T19:41:49.631452+0000","last_undegraded":"2024-09-20T19:41:49.631452+0000","last_fullsized":"2024-09-20T19:41:49.631452+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T19:40:33.239148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T19:40:33.239148+0000","last_clean_scrub_stamp":"2024-09-20T19:40:33.239148+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T01:24:39.025082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691848,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31552,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713600,"statfs":{"total":95995035648,"available":95962726400,"internally_reserved":0,"allocated":372736,"data_stored":142913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27392,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717760,"statfs":{"total":95995035648,"available":95966986240,"internally_reserved":0,"allocated":372736,"data_stored":142913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117011,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":966656,"data_stored":733281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378648,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27392,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717760,"statfs":{"total":95995035648,"available":95966986240,"internally_reserved":0,"allocated":372736,"data_stored":142913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542175,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":966656,"data_stored":733281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705700,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":966656,"data_stored":733281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T19:42:20.826 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-20T19:42:20.827 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-20T19:42:20.827 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-20T19:42:20.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T19:42:21.073 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:21.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:21 smithi114 ceph-mon[28260]: from='client.14510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:42:21.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:21 smithi114 ceph-mon[28260]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.479+0000 7f20774e2640 1 -- 172.21.15.114:0/473480848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2070107930 msgr2=0x7f2070107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.479+0000 7f20774e2640 1 --2- 172.21.15.114:0/473480848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2070107930 0x7f2070107d10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f206c009880 tx=0x7f206c02a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.479+0000 7f20774e2640 1 -- 172.21.15.114:0/473480848 shutdown_connections 2024-09-20T19:42:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.479+0000 7f20774e2640 1 --2- 172.21.15.114:0/473480848 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2070101930 0x7f2070101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.479+0000 7f20774e2640 1 --2- 172.21.15.114:0/473480848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2070107930 0x7f2070107d10 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.479+0000 7f20774e2640 1 -- 172.21.15.114:0/473480848 >> 172.21.15.114:0/473480848 conn(0x7f20700fd630 msgr2=0x7f20700ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.479+0000 7f20774e2640 1 -- 172.21.15.114:0/473480848 shutdown_connections 2024-09-20T19:42:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.479+0000 7f20774e2640 1 -- 172.21.15.114:0/473480848 wait complete. 2024-09-20T19:42:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.480+0000 7f20774e2640 1 Processor -- start 2024-09-20T19:42:21.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.480+0000 7f20774e2640 1 -- start start 2024-09-20T19:42:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.480+0000 7f20774e2640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2070101930 0x7f207019d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.480+0000 7f20774e2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2070107930 0x7f207019db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.480+0000 7f20774e2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f207010ede0 con 0x7f2070107930 2024-09-20T19:42:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.480+0000 7f20774e2640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f207010ec60 con 0x7f2070101930 2024-09-20T19:42:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.480+0000 7f2074a7d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2070107930 0x7f207019db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.480+0000 7f2074a7d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2070107930 0x7f207019db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42730/0 (socket says 172.21.15.114:42730) 2024-09-20T19:42:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.480+0000 7f2074a7d640 1 -- 172.21.15.114:0/895458385 learned_addr learned my addr 172.21.15.114:0/895458385 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.480+0000 7f207527e640 1 --2- 172.21.15.114:0/895458385 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2070101930 0x7f207019d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.480+0000 7f2074a7d640 1 -- 172.21.15.114:0/895458385 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2070101930 msgr2=0x7f207019d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.480+0000 7f2074a7d640 1 --2- 172.21.15.114:0/895458385 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2070101930 0x7f207019d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.480+0000 7f2074a7d640 1 -- 172.21.15.114:0/895458385 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20701977d0 con 0x7f2070107930 2024-09-20T19:42:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.480+0000 7f2074a7d640 1 --2- 172.21.15.114:0/895458385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2070107930 0x7f207019db90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f206000c8a0 tx=0x7f206000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.481+0000 7f20667fc640 1 -- 172.21.15.114:0/895458385 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f20600079b0 con 0x7f2070107930 2024-09-20T19:42:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.481+0000 7f20667fc640 1 -- 172.21.15.114:0/895458385 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2060007b50 con 0x7f2070107930 2024-09-20T19:42:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.481+0000 7f20774e2640 1 -- 172.21.15.114:0/895458385 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2070197ac0 con 0x7f2070107930 2024-09-20T19:42:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.481+0000 7f20667fc640 1 -- 172.21.15.114:0/895458385 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2060018700 con 0x7f2070107930 2024-09-20T19:42:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.481+0000 7f20774e2640 1 -- 172.21.15.114:0/895458385 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2070198050 con 0x7f2070107930 2024-09-20T19:42:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.482+0000 7f20774e2640 1 -- 172.21.15.114:0/895458385 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2070103070 con 0x7f2070107930 2024-09-20T19:42:21.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.482+0000 7f20667fc640 1 -- 172.21.15.114:0/895458385 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f2060007d00 con 0x7f2070107930 2024-09-20T19:42:21.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.483+0000 7f20667fc640 1 --2- 172.21.15.114:0/895458385 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f205c06e310 0x7f205c0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:21.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.483+0000 7f20667fc640 1 -- 172.21.15.114:0/895458385 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f206008e020 con 0x7f2070107930 2024-09-20T19:42:21.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.483+0000 7f207527e640 1 --2- 172.21.15.114:0/895458385 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f205c06e310 0x7f205c0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:21.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.483+0000 7f207527e640 1 --2- 172.21.15.114:0/895458385 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f205c06e310 0x7f205c0707d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f206c031460 tx=0x7f206c009560 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:21.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.487+0000 7f20667fc640 1 -- 172.21.15.114:0/895458385 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f2060014030 con 0x7f2070107930 2024-09-20T19:42:21.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.688+0000 7f20774e2640 1 -- 172.21.15.114:0/895458385 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f2070198d30 con 0x7f2070107930 2024-09-20T19:42:21.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.689+0000 7f20667fc640 1 -- 172.21.15.114:0/895458385 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f2070198d30 con 0x7f2070107930 2024-09-20T19:42:21.691 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:42:21.691 INFO:teuthology.orchestra.run.smithi114.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-20T19:42:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.690+0000 7f20774e2640 1 -- 172.21.15.114:0/895458385 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f205c06e310 msgr2=0x7f205c0707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.690+0000 7f20774e2640 1 --2- 172.21.15.114:0/895458385 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f205c06e310 0x7f205c0707d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f206c031460 tx=0x7f206c009560 comp rx=0 tx=0).stop 2024-09-20T19:42:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.690+0000 7f20774e2640 1 -- 172.21.15.114:0/895458385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2070107930 msgr2=0x7f207019db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.690+0000 7f20774e2640 1 --2- 172.21.15.114:0/895458385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2070107930 0x7f207019db90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f206000c8a0 tx=0x7f206000cd60 comp rx=0 tx=0).stop 2024-09-20T19:42:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.690+0000 7f20774e2640 1 -- 172.21.15.114:0/895458385 shutdown_connections 2024-09-20T19:42:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.690+0000 7f20774e2640 1 --2- 172.21.15.114:0/895458385 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f205c06e310 0x7f205c0707d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.690+0000 7f20774e2640 1 --2- 172.21.15.114:0/895458385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2070107930 0x7f207019db90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.690+0000 7f20774e2640 1 --2- 172.21.15.114:0/895458385 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2070101930 0x7f207019d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.690+0000 7f20774e2640 1 -- 172.21.15.114:0/895458385 >> 172.21.15.114:0/895458385 conn(0x7f20700fd630 msgr2=0x7f20700fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.691+0000 7f20774e2640 1 -- 172.21.15.114:0/895458385 shutdown_connections 2024-09-20T19:42:21.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:21.691+0000 7f20774e2640 1 -- 172.21.15.114:0/895458385 wait complete. 2024-09-20T19:42:21.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:21 smithi160 ceph-mon[35859]: from='client.14510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T19:42:21.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:21 smithi160 ceph-mon[35859]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:22.396 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-20T19:42:22.396 INFO:tasks.cephadm:Setup complete, yielding 2024-09-20T19:42:22.396 INFO:teuthology.run_tasks:Running task print... 2024-09-20T19:42:22.406 INFO:teuthology.task.print:**** done end installing squid cephadm ... 2024-09-20T19:42:22.406 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T19:42:22.427 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-09-20T19:42:22.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-09-20T19:42:22.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:22 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/895458385' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:42:22.674 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:22.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:22 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/895458385' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T19:42:23.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.066+0000 7f8be8d8e640 1 -- 172.21.15.114:0/4270931782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8be4101970 msgr2=0x7f8be4101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:23.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.066+0000 7f8be8d8e640 1 --2- 172.21.15.114:0/4270931782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8be4101970 0x7f8be4101d70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4009880 tx=0x7f8bd402a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:23.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.066+0000 7f8be8d8e640 1 -- 172.21.15.114:0/4270931782 shutdown_connections 2024-09-20T19:42:23.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.066+0000 7f8be8d8e640 1 --2- 172.21.15.114:0/4270931782 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8be4102b70 0x7f8be4102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:23.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.066+0000 7f8be8d8e640 1 --2- 172.21.15.114:0/4270931782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8be4101970 0x7f8be4101d70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:23.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.066+0000 7f8be8d8e640 1 -- 172.21.15.114:0/4270931782 >> 172.21.15.114:0/4270931782 conn(0x7f8be40fd140 msgr2=0x7f8be40ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:23.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.066+0000 7f8be8d8e640 1 -- 172.21.15.114:0/4270931782 shutdown_connections 2024-09-20T19:42:23.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.066+0000 7f8be8d8e640 1 -- 172.21.15.114:0/4270931782 wait complete. 2024-09-20T19:42:23.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.067+0000 7f8be8d8e640 1 Processor -- start 2024-09-20T19:42:23.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.067+0000 7f8be8d8e640 1 -- start start 2024-09-20T19:42:23.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.067+0000 7f8be8d8e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8be4101970 0x7f8be41974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:23.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.067+0000 7f8be8d8e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8be4102b70 0x7f8be4197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:23.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.067+0000 7f8be8d8e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8be410a810 con 0x7f8be4102b70 2024-09-20T19:42:23.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.067+0000 7f8be8d8e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8be410a690 con 0x7f8be4101970 2024-09-20T19:42:23.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.067+0000 7f8bdbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8be4102b70 0x7f8be4197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:23.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.067+0000 7f8bdbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8be4102b70 0x7f8be4197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42752/0 (socket says 172.21.15.114:42752) 2024-09-20T19:42:23.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.067+0000 7f8bdbfff640 1 -- 172.21.15.114:0/2552705490 learned_addr learned my addr 172.21.15.114:0/2552705490 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:23.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.067+0000 7f8be259c640 1 --2- 172.21.15.114:0/2552705490 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8be4101970 0x7f8be41974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:23.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.067+0000 7f8bdbfff640 1 -- 172.21.15.114:0/2552705490 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8be4101970 msgr2=0x7f8be41974e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:23.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.067+0000 7f8bdbfff640 1 --2- 172.21.15.114:0/2552705490 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8be4101970 0x7f8be41974e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:23.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.067+0000 7f8bdbfff640 1 -- 172.21.15.114:0/2552705490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8be41980a0 con 0x7f8be4102b70 2024-09-20T19:42:23.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.067+0000 7f8be259c640 1 --2- 172.21.15.114:0/2552705490 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8be4101970 0x7f8be41974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-20T19:42:23.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.067+0000 7f8bdbfff640 1 --2- 172.21.15.114:0/2552705490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8be4102b70 0x7f8be4197a20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc00b420 tx=0x7f8bcc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:23.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.068+0000 7f8bdb7fe640 1 -- 172.21.15.114:0/2552705490 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8bcc011630 con 0x7f8be4102b70 2024-09-20T19:42:23.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.068+0000 7f8bdb7fe640 1 -- 172.21.15.114:0/2552705490 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8bcc016070 con 0x7f8be4102b70 2024-09-20T19:42:23.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.068+0000 7f8be8d8e640 1 -- 172.21.15.114:0/2552705490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8be419cae0 con 0x7f8be4102b70 2024-09-20T19:42:23.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.068+0000 7f8bdb7fe640 1 -- 172.21.15.114:0/2552705490 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8bcc010d40 con 0x7f8be4102b70 2024-09-20T19:42:23.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.068+0000 7f8be8d8e640 1 -- 172.21.15.114:0/2552705490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8be419d0f0 con 0x7f8be4102b70 2024-09-20T19:42:23.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.069+0000 7f8be8d8e640 1 -- 172.21.15.114:0/2552705490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8be4101d70 con 0x7f8be4102b70 2024-09-20T19:42:23.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.069+0000 7f8bdb7fe640 1 -- 172.21.15.114:0/2552705490 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f8bcc01b460 con 0x7f8be4102b70 2024-09-20T19:42:23.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.070+0000 7f8bdb7fe640 1 --2- 172.21.15.114:0/2552705490 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8bd006e2b0 0x7f8bd0070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:23.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.070+0000 7f8bdb7fe640 1 -- 172.21.15.114:0/2552705490 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f8bcc08d730 con 0x7f8be4102b70 2024-09-20T19:42:23.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.070+0000 7f8be259c640 1 --2- 172.21.15.114:0/2552705490 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8bd006e2b0 0x7f8bd0070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:23.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.070+0000 7f8be259c640 1 --2- 172.21.15.114:0/2552705490 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8bd006e2b0 0x7f8bd0070770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8bd40096c0 tx=0x7f8bd4009650 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:23.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.074+0000 7f8bdb7fe640 1 -- 172.21.15.114:0/2552705490 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f8bcc05b3d0 con 0x7f8be4102b70 2024-09-20T19:42:23.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.241+0000 7f8be8d8e640 1 -- 172.21.15.114:0/2552705490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) -- 0x7f8be4062df0 con 0x7f8be4102b70 2024-09-20T19:42:23.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.245+0000 7f8bdb7fe640 1 -- 172.21.15.114:0/2552705490 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v41)=0 v41) ==== 143+0+0 (secure 0 0 0) 0x7f8bcc05f080 con 0x7f8be4102b70 2024-09-20T19:42:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.247+0000 7f8be8d8e640 1 -- 172.21.15.114:0/2552705490 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8bd006e2b0 msgr2=0x7f8bd0070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.247+0000 7f8be8d8e640 1 --2- 172.21.15.114:0/2552705490 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8bd006e2b0 0x7f8bd0070770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8bd40096c0 tx=0x7f8bd4009650 comp rx=0 tx=0).stop 2024-09-20T19:42:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.247+0000 7f8be8d8e640 1 -- 172.21.15.114:0/2552705490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8be4102b70 msgr2=0x7f8be4197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:23.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.247+0000 7f8be8d8e640 1 --2- 172.21.15.114:0/2552705490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8be4102b70 0x7f8be4197a20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc00b420 tx=0x7f8bcc00b8e0 comp rx=0 tx=0).stop 2024-09-20T19:42:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.247+0000 7f8be8d8e640 1 -- 172.21.15.114:0/2552705490 shutdown_connections 2024-09-20T19:42:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.247+0000 7f8be8d8e640 1 --2- 172.21.15.114:0/2552705490 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8bd006e2b0 0x7f8bd0070770 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.247+0000 7f8be8d8e640 1 --2- 172.21.15.114:0/2552705490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8be4102b70 0x7f8be4197a20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.247+0000 7f8be8d8e640 1 --2- 172.21.15.114:0/2552705490 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8be4101970 0x7f8be41974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.247+0000 7f8be8d8e640 1 -- 172.21.15.114:0/2552705490 >> 172.21.15.114:0/2552705490 conn(0x7f8be40fd140 msgr2=0x7f8be4106030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.248+0000 7f8be8d8e640 1 -- 172.21.15.114:0/2552705490 shutdown_connections 2024-09-20T19:42:23.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:23.248+0000 7f8be8d8e640 1 -- 172.21.15.114:0/2552705490 wait complete. 2024-09-20T19:42:23.614 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:23 smithi114 ceph-mon[28260]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:23.615 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:23 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/2552705490' entity='client.admin' 2024-09-20T19:42:23.615 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:23 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:42:23.615 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:23 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:42:23.615 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:23 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:42:23.615 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:23 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:23.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:23 smithi160 ceph-mon[35859]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:23.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:23 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/2552705490' entity='client.admin' 2024-09-20T19:42:23.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:23 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:42:23.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:23 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:42:23.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:23 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:42:23.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:23 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:23.917 INFO:teuthology.run_tasks:Running task print... 2024-09-20T19:42:23.933 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-09-20T19:42:23.933 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T19:42:23.973 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-09-20T19:42:23.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- bash -c 'ceph orch status' 2024-09-20T19:42:24.218 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:24.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.695+0000 7f1cfd11c640 1 -- 172.21.15.114:0/2393222578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8100880 msgr2=0x7f1cf8100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:24.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.695+0000 7f1cfd11c640 1 --2- 172.21.15.114:0/2393222578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8100880 0x7f1cf8100c80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009880 tx=0x7f1ce802a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:24.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.695+0000 7f1cfd11c640 1 -- 172.21.15.114:0/2393222578 shutdown_connections 2024-09-20T19:42:24.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.695+0000 7f1cfd11c640 1 --2- 172.21.15.114:0/2393222578 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cf8101a80 0x7f1cf8101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:24.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.695+0000 7f1cfd11c640 1 --2- 172.21.15.114:0/2393222578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8100880 0x7f1cf8100c80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.695+0000 7f1cfd11c640 1 -- 172.21.15.114:0/2393222578 >> 172.21.15.114:0/2393222578 conn(0x7f1cf80fc010 msgr2=0x7f1cf80fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.695+0000 7f1cfd11c640 1 -- 172.21.15.114:0/2393222578 shutdown_connections 2024-09-20T19:42:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.696+0000 7f1cfd11c640 1 -- 172.21.15.114:0/2393222578 wait complete. 2024-09-20T19:42:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.696+0000 7f1cfd11c640 1 Processor -- start 2024-09-20T19:42:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.697+0000 7f1cfd11c640 1 -- start start 2024-09-20T19:42:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.697+0000 7f1cfd11c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8100880 0x7f1cf806ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.697+0000 7f1cfd11c640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cf8101a80 0x7f1cf819c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.697+0000 7f1cfd11c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf8109720 con 0x7f1cf8100880 2024-09-20T19:42:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.697+0000 7f1cfd11c640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf81095a0 con 0x7f1cf8101a80 2024-09-20T19:42:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.697+0000 7f1cf6d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8100880 0x7f1cf806ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.697+0000 7f1cf6d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8100880 0x7f1cf806ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42774/0 (socket says 172.21.15.114:42774) 2024-09-20T19:42:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.697+0000 7f1cf6d9d640 1 -- 172.21.15.114:0/1862517024 learned_addr learned my addr 172.21.15.114:0/1862517024 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.697+0000 7f1cf6d9d640 1 -- 172.21.15.114:0/1862517024 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cf8101a80 msgr2=0x7f1cf819c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.697+0000 7f1cf659c640 1 --2- 172.21.15.114:0/1862517024 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cf8101a80 0x7f1cf819c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:24.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.697+0000 7f1cf6d9d640 1 --2- 172.21.15.114:0/1862517024 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cf8101a80 0x7f1cf819c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:24.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.697+0000 7f1cf6d9d640 1 -- 172.21.15.114:0/1862517024 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf819c820 con 0x7f1cf8100880 2024-09-20T19:42:24.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.697+0000 7f1cf6d9d640 1 --2- 172.21.15.114:0/1862517024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8100880 0x7f1cf806ae30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8006010 tx=0x7f1ce80048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:24.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.697+0000 7f1cdffff640 1 -- 172.21.15.114:0/1862517024 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f1ce803f070 con 0x7f1cf8100880 2024-09-20T19:42:24.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.698+0000 7f1cdffff640 1 -- 172.21.15.114:0/1862517024 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1ce80044d0 con 0x7f1cf8100880 2024-09-20T19:42:24.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.698+0000 7f1cfd11c640 1 -- 172.21.15.114:0/1862517024 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf819cab0 con 0x7f1cf8100880 2024-09-20T19:42:24.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.698+0000 7f1cdffff640 1 -- 172.21.15.114:0/1862517024 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f1ce8035030 con 0x7f1cf8100880 2024-09-20T19:42:24.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.698+0000 7f1cfd11c640 1 -- 172.21.15.114:0/1862517024 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf819d060 con 0x7f1cf8100880 2024-09-20T19:42:24.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.698+0000 7f1cfd11c640 1 -- 172.21.15.114:0/1862517024 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cf8100c80 con 0x7f1cf8100880 2024-09-20T19:42:24.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.699+0000 7f1cdffff640 1 -- 172.21.15.114:0/1862517024 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f1ce8042070 con 0x7f1cf8100880 2024-09-20T19:42:24.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.700+0000 7f1cdffff640 1 --2- 172.21.15.114:0/1862517024 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f1ce006e2b0 0x7f1ce0070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:24.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.700+0000 7f1cdffff640 1 -- 172.21.15.114:0/1862517024 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f1ce80aeb70 con 0x7f1cf8100880 2024-09-20T19:42:24.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.700+0000 7f1cf659c640 1 --2- 172.21.15.114:0/1862517024 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f1ce006e2b0 0x7f1ce0070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:24.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.700+0000 7f1cf659c640 1 --2- 172.21.15.114:0/1862517024 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f1ce006e2b0 0x7f1ce0070770 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1cf806abb0 tx=0x7f1cec008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:24.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.704+0000 7f1cdffff640 1 -- 172.21.15.114:0/1862517024 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f1ce807c780 con 0x7f1cf8100880 2024-09-20T19:42:24.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.883+0000 7f1cfd11c640 1 -- 172.21.15.114:0/1862517024 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f1cf8106000 con 0x7f1ce006e2b0 2024-09-20T19:42:24.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.884+0000 7f1cdffff640 1 -- 172.21.15.114:0/1862517024 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f1cf8106000 con 0x7f1ce006e2b0 2024-09-20T19:42:24.885 INFO:teuthology.orchestra.run.smithi114.stdout:Backend: cephadm 2024-09-20T19:42:24.886 INFO:teuthology.orchestra.run.smithi114.stdout:Available: Yes 2024-09-20T19:42:24.886 INFO:teuthology.orchestra.run.smithi114.stdout:Paused: No 2024-09-20T19:42:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.886+0000 7f1cfd11c640 1 -- 172.21.15.114:0/1862517024 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f1ce006e2b0 msgr2=0x7f1ce0070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.886+0000 7f1cfd11c640 1 --2- 172.21.15.114:0/1862517024 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f1ce006e2b0 0x7f1ce0070770 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1cf806abb0 tx=0x7f1cec008040 comp rx=0 tx=0).stop 2024-09-20T19:42:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.886+0000 7f1cfd11c640 1 -- 172.21.15.114:0/1862517024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8100880 msgr2=0x7f1cf806ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.886+0000 7f1cfd11c640 1 --2- 172.21.15.114:0/1862517024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8100880 0x7f1cf806ae30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8006010 tx=0x7f1ce80048f0 comp rx=0 tx=0).stop 2024-09-20T19:42:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.886+0000 7f1cfd11c640 1 -- 172.21.15.114:0/1862517024 shutdown_connections 2024-09-20T19:42:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.886+0000 7f1cfd11c640 1 --2- 172.21.15.114:0/1862517024 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f1ce006e2b0 0x7f1ce0070770 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.886+0000 7f1cfd11c640 1 --2- 172.21.15.114:0/1862517024 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1cf8101a80 0x7f1cf819c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.886+0000 7f1cfd11c640 1 --2- 172.21.15.114:0/1862517024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1cf8100880 0x7f1cf806ae30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.886+0000 7f1cfd11c640 1 -- 172.21.15.114:0/1862517024 >> 172.21.15.114:0/1862517024 conn(0x7f1cf80fc010 msgr2=0x7f1cf80fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.886+0000 7f1cfd11c640 1 -- 172.21.15.114:0/1862517024 shutdown_connections 2024-09-20T19:42:24.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:24.886+0000 7f1cfd11c640 1 -- 172.21.15.114:0/1862517024 wait complete. 2024-09-20T19:42:25.421 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:25 smithi114 ceph-mon[28260]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:25.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- bash -c 'ceph orch ps' 2024-09-20T19:42:25.696 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:25.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:25 smithi160 ceph-mon[35859]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:26.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.169+0000 7f53d2ca1640 1 -- 172.21.15.114:0/435519814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc101970 msgr2=0x7f53cc101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:26.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.169+0000 7f53d2ca1640 1 --2- 172.21.15.114:0/435519814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc101970 0x7f53cc101d70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f53bc009c20 tx=0x7f53bc02a970 comp rx=0 tx=0).stop 2024-09-20T19:42:26.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.169+0000 7f53d2ca1640 1 -- 172.21.15.114:0/435519814 shutdown_connections 2024-09-20T19:42:26.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.169+0000 7f53d2ca1640 1 --2- 172.21.15.114:0/435519814 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53cc102b70 0x7f53cc102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:26.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.169+0000 7f53d2ca1640 1 --2- 172.21.15.114:0/435519814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc101970 0x7f53cc101d70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:26.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.169+0000 7f53d2ca1640 1 -- 172.21.15.114:0/435519814 >> 172.21.15.114:0/435519814 conn(0x7f53cc0fd100 msgr2=0x7f53cc0ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:26.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.169+0000 7f53d2ca1640 1 -- 172.21.15.114:0/435519814 shutdown_connections 2024-09-20T19:42:26.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.169+0000 7f53d2ca1640 1 -- 172.21.15.114:0/435519814 wait complete. 2024-09-20T19:42:26.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.170+0000 7f53d2ca1640 1 Processor -- start 2024-09-20T19:42:26.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.170+0000 7f53d2ca1640 1 -- start start 2024-09-20T19:42:26.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.170+0000 7f53d2ca1640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53cc101970 0x7f53cc075eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.170+0000 7f53d2ca1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc102b70 0x7f53cc0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.170+0000 7f53d2ca1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc10a810 con 0x7f53cc102b70 2024-09-20T19:42:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.170+0000 7f53d2ca1640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc10a690 con 0x7f53cc101970 2024-09-20T19:42:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.170+0000 7f53cbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc102b70 0x7f53cc0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.170+0000 7f53cbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc102b70 0x7f53cc0745e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45214/0 (socket says 172.21.15.114:45214) 2024-09-20T19:42:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.170+0000 7f53cbfff640 1 -- 172.21.15.114:0/3343464018 learned_addr learned my addr 172.21.15.114:0/3343464018 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.170+0000 7f53d0a3d640 1 --2- 172.21.15.114:0/3343464018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53cc101970 0x7f53cc075eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.171+0000 7f53cbfff640 1 -- 172.21.15.114:0/3343464018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53cc101970 msgr2=0x7f53cc075eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.171+0000 7f53cbfff640 1 --2- 172.21.15.114:0/3343464018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53cc101970 0x7f53cc075eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.171+0000 7f53cbfff640 1 -- 172.21.15.114:0/3343464018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53cc074b20 con 0x7f53cc102b70 2024-09-20T19:42:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.171+0000 7f53cbfff640 1 --2- 172.21.15.114:0/3343464018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc102b70 0x7f53cc0745e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f53b400b420 tx=0x7f53b400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:26.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.171+0000 7f53c9ffb640 1 -- 172.21.15.114:0/3343464018 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f53b4011630 con 0x7f53cc102b70 2024-09-20T19:42:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.171+0000 7f53d2ca1640 1 -- 172.21.15.114:0/3343464018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53cc074e10 con 0x7f53cc102b70 2024-09-20T19:42:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.171+0000 7f53c9ffb640 1 -- 172.21.15.114:0/3343464018 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f53b4016070 con 0x7f53cc102b70 2024-09-20T19:42:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.171+0000 7f53c9ffb640 1 -- 172.21.15.114:0/3343464018 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f53b4010d40 con 0x7f53cc102b70 2024-09-20T19:42:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.171+0000 7f53d2ca1640 1 -- 172.21.15.114:0/3343464018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53cc075420 con 0x7f53cc102b70 2024-09-20T19:42:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.172+0000 7f53d2ca1640 1 -- 172.21.15.114:0/3343464018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53cc101df0 con 0x7f53cc102b70 2024-09-20T19:42:26.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.173+0000 7f53c9ffb640 1 -- 172.21.15.114:0/3343464018 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f53b401b460 con 0x7f53cc102b70 2024-09-20T19:42:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.173+0000 7f53c9ffb640 1 --2- 172.21.15.114:0/3343464018 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f53b806e2b0 0x7f53b8070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.173+0000 7f53c9ffb640 1 -- 172.21.15.114:0/3343464018 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f53b408d5d0 con 0x7f53cc102b70 2024-09-20T19:42:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.173+0000 7f53d0a3d640 1 --2- 172.21.15.114:0/3343464018 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f53b806e2b0 0x7f53b8070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:26.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.174+0000 7f53d0a3d640 1 --2- 172.21.15.114:0/3343464018 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f53b806e2b0 0x7f53b8070770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f53bc000f80 tx=0x7f53bc03c040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:26.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.178+0000 7f53c9ffb640 1 -- 172.21.15.114:0/3343464018 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f53b405b270 con 0x7f53cc102b70 2024-09-20T19:42:26.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.352+0000 7f53d2ca1640 1 -- 172.21.15.114:0/3343464018 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f53cc062630 con 0x7f53b806e2b0 2024-09-20T19:42:26.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.360+0000 7f53c9ffb640 1 -- 172.21.15.114:0/3343464018 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3120 (secure 0 0 0) 0x7f53cc062630 con 0x7f53b806e2b0 2024-09-20T19:42:26.362 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:42:26.362 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (3m) 111s ago 4m 26.3M - 0.25.0 c8568f914cd2 d19ec799e98c 2024-09-20T19:42:26.362 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (5m) 111s ago 5m 7763k - 19.1.1-369-ged8ee0f2 6acb0a031eab ddd91f706b5f 2024-09-20T19:42:26.362 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi160 smithi160 running (4m) 42s ago 4m 5268k - 19.1.1-369-ged8ee0f2 6acb0a031eab 8ce900c6456f 2024-09-20T19:42:26.362 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (5m) 111s ago 5m 6895k - 19.1.1-369-ged8ee0f2 6acb0a031eab 0e32ab0eeecd 2024-09-20T19:42:26.362 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi160 smithi160 running (4m) 42s ago 4m 6891k - 19.1.1-369-ged8ee0f2 6acb0a031eab d858e40463d6 2024-09-20T19:42:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (3m) 111s ago 4m 73.2M - 9.4.12 2bacad6d85d8 cd034302ef4b 2024-09-20T19:42:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.rvhgji smithi114 *:9283,8765,8443 running (6m) 111s ago 6m 490M - 19.1.1-369-ged8ee0f2 6acb0a031eab 59997c1e986a 2024-09-20T19:42:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi160.bspiak smithi160 *:8443,9283,8765 running (4m) 42s ago 4m 440M - 19.1.1-369-ged8ee0f2 6acb0a031eab 50306543ba66 2024-09-20T19:42:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (6m) 111s ago 6m 46.0M 2048M 19.1.1-369-ged8ee0f2 6acb0a031eab ed7f013f2513 2024-09-20T19:42:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi160 smithi160 running (4m) 42s ago 4m 40.6M 2048M 19.1.1-369-ged8ee0f2 6acb0a031eab 961cc10b05cc 2024-09-20T19:42:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (5m) 111s ago 5m 16.4M - 1.5.0 0da6a335fe13 6542a55ff841 2024-09-20T19:42:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi160 smithi160 *:9100 running (4m) 42s ago 4m 16.4M - 1.5.0 0da6a335fe13 ff98c99cbe97 2024-09-20T19:42:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (3m) 111s ago 3m 39.2M 3946M 19.1.1-369-ged8ee0f2 6acb0a031eab 6f0510eaf505 2024-09-20T19:42:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (2m) 111s ago 2m 44.8M 3946M 19.1.1-369-ged8ee0f2 6acb0a031eab 98b6e5128717 2024-09-20T19:42:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (2m) 111s ago 2m 37.5M 3946M 19.1.1-369-ged8ee0f2 6acb0a031eab dda945a7e7ca 2024-09-20T19:42:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi160 running (100s) 42s ago 100s 47.2M 4970M 19.1.1-369-ged8ee0f2 6acb0a031eab 9facdfb1a166 2024-09-20T19:42:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi160 running (73s) 42s ago 73s 39.9M 4970M 19.1.1-369-ged8ee0f2 6acb0a031eab e15086d3d575 2024-09-20T19:42:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi160 running (47s) 42s ago 47s 14.2M 4970M 19.1.1-369-ged8ee0f2 6acb0a031eab 7f90f136e99a 2024-09-20T19:42:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (3m) 111s ago 4m 42.7M - 2.43.0 a07b618ecd1d 733ef913ac0b 2024-09-20T19:42:26.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.362+0000 7f53d2ca1640 1 -- 172.21.15.114:0/3343464018 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f53b806e2b0 msgr2=0x7f53b8070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:26.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.362+0000 7f53d2ca1640 1 --2- 172.21.15.114:0/3343464018 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f53b806e2b0 0x7f53b8070770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f53bc000f80 tx=0x7f53bc03c040 comp rx=0 tx=0).stop 2024-09-20T19:42:26.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.362+0000 7f53d2ca1640 1 -- 172.21.15.114:0/3343464018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc102b70 msgr2=0x7f53cc0745e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:26.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.362+0000 7f53d2ca1640 1 --2- 172.21.15.114:0/3343464018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc102b70 0x7f53cc0745e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f53b400b420 tx=0x7f53b400b8e0 comp rx=0 tx=0).stop 2024-09-20T19:42:26.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.362+0000 7f53d2ca1640 1 -- 172.21.15.114:0/3343464018 shutdown_connections 2024-09-20T19:42:26.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.362+0000 7f53d2ca1640 1 --2- 172.21.15.114:0/3343464018 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f53b806e2b0 0x7f53b8070770 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:26.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.362+0000 7f53d2ca1640 1 --2- 172.21.15.114:0/3343464018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc102b70 0x7f53cc0745e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:26.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.362+0000 7f53d2ca1640 1 --2- 172.21.15.114:0/3343464018 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f53cc101970 0x7f53cc075eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:26.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.362+0000 7f53d2ca1640 1 -- 172.21.15.114:0/3343464018 >> 172.21.15.114:0/3343464018 conn(0x7f53cc0fd100 msgr2=0x7f53cc1060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:26.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.363+0000 7f53d2ca1640 1 -- 172.21.15.114:0/3343464018 shutdown_connections 2024-09-20T19:42:26.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:26.363+0000 7f53d2ca1640 1 -- 172.21.15.114:0/3343464018 wait complete. 2024-09-20T19:42:26.661 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:26 smithi114 ceph-mon[28260]: from='client.14522 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:26.661 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:26 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:42:26.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:26 smithi160 ceph-mon[35859]: from='client.14522 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:26.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:26 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:42:26.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- bash -c 'ceph orch ls' 2024-09-20T19:42:27.217 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:27.609 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:27 smithi114 ceph-mon[28260]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:27.609 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:27 smithi114 ceph-mon[28260]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:27.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.628+0000 7fd944b6e640 1 -- 172.21.15.114:0/2982696260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd940101a80 msgr2=0x7fd940101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:27.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.628+0000 7fd944b6e640 1 --2- 172.21.15.114:0/2982696260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd940101a80 0x7fd940101f00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd928009880 tx=0x7fd92802a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:27.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.629+0000 7fd944b6e640 1 -- 172.21.15.114:0/2982696260 shutdown_connections 2024-09-20T19:42:27.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.629+0000 7fd944b6e640 1 --2- 172.21.15.114:0/2982696260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd940101a80 0x7fd940101f00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:27.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.629+0000 7fd944b6e640 1 --2- 172.21.15.114:0/2982696260 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd940100880 0x7fd940100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:27.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.629+0000 7fd944b6e640 1 -- 172.21.15.114:0/2982696260 >> 172.21.15.114:0/2982696260 conn(0x7fd9400fc010 msgr2=0x7fd9400fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:27.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.629+0000 7fd944b6e640 1 -- 172.21.15.114:0/2982696260 shutdown_connections 2024-09-20T19:42:27.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.629+0000 7fd944b6e640 1 -- 172.21.15.114:0/2982696260 wait complete. 2024-09-20T19:42:27.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.629+0000 7fd944b6e640 1 Processor -- start 2024-09-20T19:42:27.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.629+0000 7fd944b6e640 1 -- start start 2024-09-20T19:42:27.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.629+0000 7fd944b6e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd940100880 0x7fd940197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:27.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd944b6e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd940101a80 0x7fd940197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:27.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd944b6e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd940109740 con 0x7fd940100880 2024-09-20T19:42:27.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd944b6e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd9401095c0 con 0x7fd940101a80 2024-09-20T19:42:27.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd93e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd940100880 0x7fd940197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:27.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd93e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd940100880 0x7fd940197500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45230/0 (socket says 172.21.15.114:45230) 2024-09-20T19:42:27.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd93e59c640 1 -- 172.21.15.114:0/2701498669 learned_addr learned my addr 172.21.15.114:0/2701498669 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:27.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd937fff640 1 --2- 172.21.15.114:0/2701498669 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd940101a80 0x7fd940197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:27.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd93e59c640 1 -- 172.21.15.114:0/2701498669 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd940101a80 msgr2=0x7fd940197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:27.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd93e59c640 1 --2- 172.21.15.114:0/2701498669 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd940101a80 0x7fd940197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:27.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd93e59c640 1 -- 172.21.15.114:0/2701498669 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9401980c0 con 0x7fd940100880 2024-09-20T19:42:27.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd93e59c640 1 --2- 172.21.15.114:0/2701498669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd940100880 0x7fd940197500 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd93000b3d0 tx=0x7fd93000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:27.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd9377fe640 1 -- 172.21.15.114:0/2701498669 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fd930015020 con 0x7fd940100880 2024-09-20T19:42:27.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd9377fe640 1 -- 172.21.15.114:0/2701498669 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd930011af0 con 0x7fd940100880 2024-09-20T19:42:27.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd944b6e640 1 -- 172.21.15.114:0/2701498669 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd94019cb00 con 0x7fd940100880 2024-09-20T19:42:27.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd9377fe640 1 -- 172.21.15.114:0/2701498669 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fd930010da0 con 0x7fd940100880 2024-09-20T19:42:27.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.630+0000 7fd944b6e640 1 -- 172.21.15.114:0/2701498669 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd94019d060 con 0x7fd940100880 2024-09-20T19:42:27.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.631+0000 7fd944b6e640 1 -- 172.21.15.114:0/2701498669 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd940100c80 con 0x7fd940100880 2024-09-20T19:42:27.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.632+0000 7fd9377fe640 1 -- 172.21.15.114:0/2701498669 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fd930018020 con 0x7fd940100880 2024-09-20T19:42:27.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.632+0000 7fd9377fe640 1 --2- 172.21.15.114:0/2701498669 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd92c06e2b0 0x7fd92c070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:27.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.632+0000 7fd9377fe640 1 -- 172.21.15.114:0/2701498669 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7fd93008d930 con 0x7fd940100880 2024-09-20T19:42:27.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.632+0000 7fd937fff640 1 --2- 172.21.15.114:0/2701498669 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd92c06e2b0 0x7fd92c070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:27.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.633+0000 7fd937fff640 1 --2- 172.21.15.114:0/2701498669 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd92c06e2b0 0x7fd92c070770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd928000c00 tx=0x7fd9280058d0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:27.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.637+0000 7fd9377fe640 1 -- 172.21.15.114:0/2701498669 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fd93001f020 con 0x7fd940100880 2024-09-20T19:42:27.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:27 smithi160 ceph-mon[35859]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:27.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:27 smithi160 ceph-mon[35859]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:27.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.812+0000 7fd944b6e640 1 -- 172.21.15.114:0/2701498669 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fd9400626f0 con 0x7fd92c06e2b0 2024-09-20T19:42:27.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.817+0000 7fd9377fe640 1 -- 172.21.15.114:0/2701498669 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+1310 (secure 0 0 0) 0x7fd9400626f0 con 0x7fd92c06e2b0 2024-09-20T19:42:27.818 INFO:teuthology.orchestra.run.smithi114.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-20T19:42:27.818 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager ?:9093,9094 1/1 112s ago 5m count:1 2024-09-20T19:42:27.818 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter 2/2 112s ago 5m * 2024-09-20T19:42:27.818 INFO:teuthology.orchestra.run.smithi114.stdout:crash 2/2 112s ago 5m * 2024-09-20T19:42:27.819 INFO:teuthology.orchestra.run.smithi114.stdout:grafana ?:3000 1/1 112s ago 5m count:1 2024-09-20T19:42:27.819 INFO:teuthology.orchestra.run.smithi114.stdout:mgr 2/2 112s ago 5m count:2 2024-09-20T19:42:27.819 INFO:teuthology.orchestra.run.smithi114.stdout:mon 2/2 112s ago 5m smithi114:172.21.15.114=smithi114;smithi160:172.21.15.160=smithi160;count:2 2024-09-20T19:42:27.819 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter ?:9100 2/2 112s ago 5m * 2024-09-20T19:42:27.819 INFO:teuthology.orchestra.run.smithi114.stdout:osd 6 112s ago - 2024-09-20T19:42:27.819 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus ?:9095 1/1 112s ago 5m count:1 2024-09-20T19:42:27.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.819+0000 7fd944b6e640 1 -- 172.21.15.114:0/2701498669 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd92c06e2b0 msgr2=0x7fd92c070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:27.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.819+0000 7fd944b6e640 1 --2- 172.21.15.114:0/2701498669 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd92c06e2b0 0x7fd92c070770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd928000c00 tx=0x7fd9280058d0 comp rx=0 tx=0).stop 2024-09-20T19:42:27.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.819+0000 7fd944b6e640 1 -- 172.21.15.114:0/2701498669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd940100880 msgr2=0x7fd940197500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:27.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.819+0000 7fd944b6e640 1 --2- 172.21.15.114:0/2701498669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd940100880 0x7fd940197500 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd93000b3d0 tx=0x7fd93000b890 comp rx=0 tx=0).stop 2024-09-20T19:42:27.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.819+0000 7fd944b6e640 1 -- 172.21.15.114:0/2701498669 shutdown_connections 2024-09-20T19:42:27.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.819+0000 7fd944b6e640 1 --2- 172.21.15.114:0/2701498669 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd92c06e2b0 0x7fd92c070770 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:27.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.819+0000 7fd944b6e640 1 --2- 172.21.15.114:0/2701498669 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd940101a80 0x7fd940197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:27.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.819+0000 7fd944b6e640 1 --2- 172.21.15.114:0/2701498669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd940100880 0x7fd940197500 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:27.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.819+0000 7fd944b6e640 1 -- 172.21.15.114:0/2701498669 >> 172.21.15.114:0/2701498669 conn(0x7fd9400fc010 msgr2=0x7fd9400fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:27.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.819+0000 7fd944b6e640 1 -- 172.21.15.114:0/2701498669 shutdown_connections 2024-09-20T19:42:27.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:27.819+0000 7fd944b6e640 1 -- 172.21.15.114:0/2701498669 wait complete. 2024-09-20T19:42:28.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- bash -c 'ceph orch host ls' 2024-09-20T19:42:28.733 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:29.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.144+0000 7f56417cc640 1 -- 172.21.15.114:0/2154295838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f563c1023d0 msgr2=0x7f563c1047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.144+0000 7f56417cc640 1 --2- 172.21.15.114:0/2154295838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f563c1023d0 0x7f563c1047c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5634009c50 tx=0x7f563402a9a0 comp rx=0 tx=0).stop 2024-09-20T19:42:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.144+0000 7f56417cc640 1 -- 172.21.15.114:0/2154295838 shutdown_connections 2024-09-20T19:42:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.144+0000 7f56417cc640 1 --2- 172.21.15.114:0/2154295838 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f563c104d00 0x7f563c1070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.144+0000 7f56417cc640 1 --2- 172.21.15.114:0/2154295838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f563c1023d0 0x7f563c1047c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.144+0000 7f56417cc640 1 -- 172.21.15.114:0/2154295838 >> 172.21.15.114:0/2154295838 conn(0x7f563c0fbff0 msgr2=0x7f563c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.144+0000 7f56417cc640 1 -- 172.21.15.114:0/2154295838 shutdown_connections 2024-09-20T19:42:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.144+0000 7f56417cc640 1 -- 172.21.15.114:0/2154295838 wait complete. 2024-09-20T19:42:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.145+0000 7f56417cc640 1 Processor -- start 2024-09-20T19:42:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.145+0000 7f56417cc640 1 -- start start 2024-09-20T19:42:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.145+0000 7f56417cc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f563c1023d0 0x7f563c197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.145+0000 7f56417cc640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f563c104d00 0x7f563c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.145+0000 7f56417cc640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f563c10a7c0 con 0x7f563c1023d0 2024-09-20T19:42:29.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.145+0000 7f56417cc640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f563c10a640 con 0x7f563c104d00 2024-09-20T19:42:29.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.145+0000 7f563affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f563c1023d0 0x7f563c197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:29.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.145+0000 7f563affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f563c1023d0 0x7f563c197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45240/0 (socket says 172.21.15.114:45240) 2024-09-20T19:42:29.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.145+0000 7f563affd640 1 -- 172.21.15.114:0/3191529751 learned_addr learned my addr 172.21.15.114:0/3191529751 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:29.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.146+0000 7f563a7fc640 1 --2- 172.21.15.114:0/3191529751 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f563c104d00 0x7f563c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:29.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.146+0000 7f563affd640 1 -- 172.21.15.114:0/3191529751 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f563c104d00 msgr2=0x7f563c197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:29.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.146+0000 7f563affd640 1 --2- 172.21.15.114:0/3191529751 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f563c104d00 0x7f563c197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:29.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.146+0000 7f563affd640 1 -- 172.21.15.114:0/3191529751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f563c1980e0 con 0x7f563c1023d0 2024-09-20T19:42:29.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.146+0000 7f563affd640 1 --2- 172.21.15.114:0/3191529751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f563c1023d0 0x7f563c197520 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5634009d70 tx=0x7f563402fb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:29.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.146+0000 7f5633fff640 1 -- 172.21.15.114:0/3191529751 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5634034030 con 0x7f563c1023d0 2024-09-20T19:42:29.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.146+0000 7f56417cc640 1 -- 172.21.15.114:0/3191529751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f563c19cac0 con 0x7f563c1023d0 2024-09-20T19:42:29.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.146+0000 7f5633fff640 1 -- 172.21.15.114:0/3191529751 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5634030b20 con 0x7f563c1023d0 2024-09-20T19:42:29.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.146+0000 7f5633fff640 1 -- 172.21.15.114:0/3191529751 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f56340046b0 con 0x7f563c1023d0 2024-09-20T19:42:29.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.146+0000 7f56417cc640 1 -- 172.21.15.114:0/3191529751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f563c19d050 con 0x7f563c1023d0 2024-09-20T19:42:29.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.147+0000 7f56417cc640 1 -- 172.21.15.114:0/3191529751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f563c1027b0 con 0x7f563c1023d0 2024-09-20T19:42:29.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.148+0000 7f5633fff640 1 -- 172.21.15.114:0/3191529751 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f5634037020 con 0x7f563c1023d0 2024-09-20T19:42:29.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.148+0000 7f5633fff640 1 --2- 172.21.15.114:0/3191529751 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f562806e2b0 0x7f5628070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:29.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.148+0000 7f563a7fc640 1 --2- 172.21.15.114:0/3191529751 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f562806e2b0 0x7f5628070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:29.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.148+0000 7f5633fff640 1 -- 172.21.15.114:0/3191529751 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7f56340aee60 con 0x7f563c1023d0 2024-09-20T19:42:29.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.148+0000 7f563a7fc640 1 --2- 172.21.15.114:0/3191529751 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f562806e2b0 0x7f5628070770 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5624005fd0 tx=0x7f5624005850 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:29.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.152+0000 7f5633fff640 1 -- 172.21.15.114:0/3191529751 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f563407cb00 con 0x7f563c1023d0 2024-09-20T19:42:29.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.323+0000 7f56417cc640 1 -- 172.21.15.114:0/3191529751 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f563c1056e0 con 0x7f562806e2b0 2024-09-20T19:42:29.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.325+0000 7f5633fff640 1 -- 172.21.15.114:0/3191529751 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+148 (secure 0 0 0) 0x7f563c1056e0 con 0x7f562806e2b0 2024-09-20T19:42:29.326 INFO:teuthology.orchestra.run.smithi114.stdout:HOST ADDR LABELS STATUS 2024-09-20T19:42:29.326 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114 172.21.15.114 2024-09-20T19:42:29.326 INFO:teuthology.orchestra.run.smithi114.stdout:smithi160 172.21.15.160 2024-09-20T19:42:29.327 INFO:teuthology.orchestra.run.smithi114.stdout:2 hosts in cluster 2024-09-20T19:42:29.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.327+0000 7f56417cc640 1 -- 172.21.15.114:0/3191529751 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f562806e2b0 msgr2=0x7f5628070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.327+0000 7f56417cc640 1 --2- 172.21.15.114:0/3191529751 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f562806e2b0 0x7f5628070770 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5624005fd0 tx=0x7f5624005850 comp rx=0 tx=0).stop 2024-09-20T19:42:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.327+0000 7f56417cc640 1 -- 172.21.15.114:0/3191529751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f563c1023d0 msgr2=0x7f563c197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.327+0000 7f56417cc640 1 --2- 172.21.15.114:0/3191529751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f563c1023d0 0x7f563c197520 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5634009d70 tx=0x7f563402fb90 comp rx=0 tx=0).stop 2024-09-20T19:42:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.327+0000 7f56417cc640 1 -- 172.21.15.114:0/3191529751 shutdown_connections 2024-09-20T19:42:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.327+0000 7f56417cc640 1 --2- 172.21.15.114:0/3191529751 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f562806e2b0 0x7f5628070770 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.327+0000 7f56417cc640 1 --2- 172.21.15.114:0/3191529751 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f563c104d00 0x7f563c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.327+0000 7f56417cc640 1 --2- 172.21.15.114:0/3191529751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f563c1023d0 0x7f563c197520 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.327+0000 7f56417cc640 1 -- 172.21.15.114:0/3191529751 >> 172.21.15.114:0/3191529751 conn(0x7f563c0fbff0 msgr2=0x7f563c0fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.327+0000 7f56417cc640 1 -- 172.21.15.114:0/3191529751 shutdown_connections 2024-09-20T19:42:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:29.327+0000 7f56417cc640 1 -- 172.21.15.114:0/3191529751 wait complete. 2024-09-20T19:42:29.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:29 smithi114 ceph-mon[28260]: from='client.14530 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:29.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:29 smithi114 ceph-mon[28260]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:29.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:29 smithi160 ceph-mon[35859]: from='client.14530 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:29.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:29 smithi160 ceph-mon[35859]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:29.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- bash -c 'ceph orch device ls' 2024-09-20T19:42:30.193 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:30.622 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:30 smithi114 ceph-mon[28260]: from='client.14534 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:30.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.627+0000 7faefaeb0640 1 -- 172.21.15.114:0/2747911533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef4102b90 msgr2=0x7faef4103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:30.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.627+0000 7faefaeb0640 1 --2- 172.21.15.114:0/2747911533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef4102b90 0x7faef4103010 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faee4009880 tx=0x7faee402a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:30.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.627+0000 7faefaeb0640 1 -- 172.21.15.114:0/2747911533 shutdown_connections 2024-09-20T19:42:30.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.627+0000 7faefaeb0640 1 --2- 172.21.15.114:0/2747911533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef4102b90 0x7faef4103010 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:30.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.627+0000 7faefaeb0640 1 --2- 172.21.15.114:0/2747911533 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faef4101990 0x7faef4101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:30.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.627+0000 7faefaeb0640 1 -- 172.21.15.114:0/2747911533 >> 172.21.15.114:0/2747911533 conn(0x7faef40fd140 msgr2=0x7faef40ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:30.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.627+0000 7faefaeb0640 1 -- 172.21.15.114:0/2747911533 shutdown_connections 2024-09-20T19:42:30.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.627+0000 7faefaeb0640 1 -- 172.21.15.114:0/2747911533 wait complete. 2024-09-20T19:42:30.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.628+0000 7faefaeb0640 1 Processor -- start 2024-09-20T19:42:30.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.628+0000 7faefaeb0640 1 -- start start 2024-09-20T19:42:30.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.628+0000 7faefaeb0640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faef4101990 0x7faef4075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:30.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.628+0000 7faefaeb0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef4102b90 0x7faef40745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:30.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.628+0000 7faefaeb0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faef410a850 con 0x7faef4102b90 2024-09-20T19:42:30.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.628+0000 7faefaeb0640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7faef410a6d0 con 0x7faef4101990 2024-09-20T19:42:30.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.628+0000 7faeebfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef4102b90 0x7faef40745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:30.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.628+0000 7faeebfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef4102b90 0x7faef40745e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45252/0 (socket says 172.21.15.114:45252) 2024-09-20T19:42:30.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.628+0000 7faeebfff640 1 -- 172.21.15.114:0/356970901 learned_addr learned my addr 172.21.15.114:0/356970901 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:30.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.629+0000 7faeebfff640 1 -- 172.21.15.114:0/356970901 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faef4101990 msgr2=0x7faef4075f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:30.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.629+0000 7faeebfff640 1 --2- 172.21.15.114:0/356970901 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faef4101990 0x7faef4075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:30.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.629+0000 7faeebfff640 1 -- 172.21.15.114:0/356970901 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef4074b20 con 0x7faef4102b90 2024-09-20T19:42:30.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.629+0000 7faeebfff640 1 --2- 172.21.15.114:0/356970901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef4102b90 0x7faef40745e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7faee4006010 tx=0x7faee40048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:30.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.629+0000 7faee9ffb640 1 -- 172.21.15.114:0/356970901 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7faee403f070 con 0x7faef4102b90 2024-09-20T19:42:30.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.629+0000 7faefaeb0640 1 -- 172.21.15.114:0/356970901 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef4074db0 con 0x7faef4102b90 2024-09-20T19:42:30.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.629+0000 7faee9ffb640 1 -- 172.21.15.114:0/356970901 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7faee40044d0 con 0x7faef4102b90 2024-09-20T19:42:30.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.629+0000 7faee9ffb640 1 -- 172.21.15.114:0/356970901 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7faee4035030 con 0x7faef4102b90 2024-09-20T19:42:30.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.629+0000 7faefaeb0640 1 -- 172.21.15.114:0/356970901 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef4075360 con 0x7faef4102b90 2024-09-20T19:42:30.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.630+0000 7faefaeb0640 1 -- 172.21.15.114:0/356970901 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faef4101d70 con 0x7faef4102b90 2024-09-20T19:42:30.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.631+0000 7faee9ffb640 1 -- 172.21.15.114:0/356970901 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7faee4042070 con 0x7faef4102b90 2024-09-20T19:42:30.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.631+0000 7faee9ffb640 1 --2- 172.21.15.114:0/356970901 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7faedc06e2b0 0x7faedc070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:30.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.631+0000 7faee9ffb640 1 -- 172.21.15.114:0/356970901 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7faee40aebd0 con 0x7faef4102b90 2024-09-20T19:42:30.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.631+0000 7faef8c4c640 1 --2- 172.21.15.114:0/356970901 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7faedc06e2b0 0x7faedc070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:30.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.632+0000 7faef8c4c640 1 --2- 172.21.15.114:0/356970901 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7faedc06e2b0 0x7faedc070770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faef41029f0 tx=0x7faef0009250 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:30.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.636+0000 7faee9ffb640 1 -- 172.21.15.114:0/356970901 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7faee407c870 con 0x7faef4102b90 2024-09-20T19:42:30.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:30 smithi160 ceph-mon[35859]: from='client.14534 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:30.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.809+0000 7faefaeb0640 1 -- 172.21.15.114:0/356970901 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7faef4062630 con 0x7faedc06e2b0 2024-09-20T19:42:30.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.811+0000 7faee9ffb640 1 -- 172.21.15.114:0/356970901 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+399 (secure 0 0 0) 0x7faef4062630 con 0x7faedc06e2b0 2024-09-20T19:42:30.812 INFO:teuthology.orchestra.run.smithi114.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-20T19:42:30.812 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400WB400BGN 372G No 119s ago Has a FileSystem, LVM detected 2024-09-20T19:42:30.813 INFO:teuthology.orchestra.run.smithi114.stdout:smithi160 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JY400BGN 372G No 43s ago Has a FileSystem, LVM detected 2024-09-20T19:42:30.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.813+0000 7faefaeb0640 1 -- 172.21.15.114:0/356970901 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7faedc06e2b0 msgr2=0x7faedc070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:30.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.813+0000 7faefaeb0640 1 --2- 172.21.15.114:0/356970901 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7faedc06e2b0 0x7faedc070770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faef41029f0 tx=0x7faef0009250 comp rx=0 tx=0).stop 2024-09-20T19:42:30.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.813+0000 7faefaeb0640 1 -- 172.21.15.114:0/356970901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef4102b90 msgr2=0x7faef40745e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:30.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.813+0000 7faefaeb0640 1 --2- 172.21.15.114:0/356970901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef4102b90 0x7faef40745e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7faee4006010 tx=0x7faee40048f0 comp rx=0 tx=0).stop 2024-09-20T19:42:30.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.813+0000 7faefaeb0640 1 -- 172.21.15.114:0/356970901 shutdown_connections 2024-09-20T19:42:30.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.813+0000 7faefaeb0640 1 --2- 172.21.15.114:0/356970901 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7faedc06e2b0 0x7faedc070770 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:30.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.813+0000 7faefaeb0640 1 --2- 172.21.15.114:0/356970901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef4102b90 0x7faef40745e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:30.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.813+0000 7faefaeb0640 1 --2- 172.21.15.114:0/356970901 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faef4101990 0x7faef4075f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:30.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.813+0000 7faefaeb0640 1 -- 172.21.15.114:0/356970901 >> 172.21.15.114:0/356970901 conn(0x7faef40fd140 msgr2=0x7faef4106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:30.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.813+0000 7faefaeb0640 1 -- 172.21.15.114:0/356970901 shutdown_connections 2024-09-20T19:42:30.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:30.813+0000 7faefaeb0640 1 -- 172.21.15.114:0/356970901 wait complete. 2024-09-20T19:42:31.451 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:31 smithi114 ceph-mon[28260]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:31.479 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T19:42:31.489 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-09-20T19:42:31.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-09-20T19:42:31.737 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:31.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:31 smithi160 ceph-mon[35859]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:32.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.143+0000 7fc602d7e640 1 -- 172.21.15.114:0/3876617616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5fc101a80 msgr2=0x7fc5fc101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:32.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.143+0000 7fc602d7e640 1 --2- 172.21.15.114:0/3876617616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5fc101a80 0x7fc5fc101f00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec009880 tx=0x7fc5ec02a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:32.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.143+0000 7fc602d7e640 1 -- 172.21.15.114:0/3876617616 shutdown_connections 2024-09-20T19:42:32.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.143+0000 7fc602d7e640 1 --2- 172.21.15.114:0/3876617616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5fc101a80 0x7fc5fc101f00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:32.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.143+0000 7fc602d7e640 1 --2- 172.21.15.114:0/3876617616 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc5fc100880 0x7fc5fc100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:32.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.143+0000 7fc602d7e640 1 -- 172.21.15.114:0/3876617616 >> 172.21.15.114:0/3876617616 conn(0x7fc5fc0fc010 msgr2=0x7fc5fc0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:32.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.143+0000 7fc602d7e640 1 -- 172.21.15.114:0/3876617616 shutdown_connections 2024-09-20T19:42:32.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.143+0000 7fc602d7e640 1 -- 172.21.15.114:0/3876617616 wait complete. 2024-09-20T19:42:32.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.144+0000 7fc602d7e640 1 Processor -- start 2024-09-20T19:42:32.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.144+0000 7fc602d7e640 1 -- start start 2024-09-20T19:42:32.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.144+0000 7fc602d7e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5fc100880 0x7fc5fc197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:32.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.144+0000 7fc602d7e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc5fc101a80 0x7fc5fc197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:32.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.144+0000 7fc602d7e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc5fc109740 con 0x7fc5fc100880 2024-09-20T19:42:32.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.144+0000 7fc602d7e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc5fc1095c0 con 0x7fc5fc101a80 2024-09-20T19:42:32.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.144+0000 7fc600b1a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5fc100880 0x7fc5fc197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:32.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.144+0000 7fc600b1a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5fc100880 0x7fc5fc197580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45274/0 (socket says 172.21.15.114:45274) 2024-09-20T19:42:32.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.144+0000 7fc600b1a640 1 -- 172.21.15.114:0/1497564332 learned_addr learned my addr 172.21.15.114:0/1497564332 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:32.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.145+0000 7fc600b1a640 1 -- 172.21.15.114:0/1497564332 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc5fc101a80 msgr2=0x7fc5fc197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:32.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.145+0000 7fc600b1a640 1 --2- 172.21.15.114:0/1497564332 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc5fc101a80 0x7fc5fc197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:32.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.145+0000 7fc600b1a640 1 -- 172.21.15.114:0/1497564332 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5fc198140 con 0x7fc5fc100880 2024-09-20T19:42:32.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.145+0000 7fc600b1a640 1 --2- 172.21.15.114:0/1497564332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5fc100880 0x7fc5fc197580 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc5f800b3d0 tx=0x7fc5f800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:32.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.145+0000 7fc5f1ffb640 1 -- 172.21.15.114:0/1497564332 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc5f8015020 con 0x7fc5fc100880 2024-09-20T19:42:32.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.145+0000 7fc5f1ffb640 1 -- 172.21.15.114:0/1497564332 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc5f8011af0 con 0x7fc5fc100880 2024-09-20T19:42:32.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.145+0000 7fc5f1ffb640 1 -- 172.21.15.114:0/1497564332 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc5f8010d40 con 0x7fc5fc100880 2024-09-20T19:42:32.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.145+0000 7fc602d7e640 1 -- 172.21.15.114:0/1497564332 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5fc19cb80 con 0x7fc5fc100880 2024-09-20T19:42:32.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.146+0000 7fc602d7e640 1 -- 172.21.15.114:0/1497564332 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5fc19d190 con 0x7fc5fc100880 2024-09-20T19:42:32.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.147+0000 7fc602d7e640 1 -- 172.21.15.114:0/1497564332 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5fc100c60 con 0x7fc5fc100880 2024-09-20T19:42:32.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.147+0000 7fc5f1ffb640 1 -- 172.21.15.114:0/1497564332 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fc5f8018020 con 0x7fc5fc100880 2024-09-20T19:42:32.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.147+0000 7fc5f1ffb640 1 --2- 172.21.15.114:0/1497564332 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc5e406e350 0x7fc5e4070810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:32.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.147+0000 7fc5f1ffb640 1 -- 172.21.15.114:0/1497564332 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4638+0+0 (secure 0 0 0) 0x7fc5f808d860 con 0x7fc5fc100880 2024-09-20T19:42:32.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.147+0000 7fc5f3fff640 1 --2- 172.21.15.114:0/1497564332 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc5e406e350 0x7fc5e4070810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:32.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.148+0000 7fc5f3fff640 1 --2- 172.21.15.114:0/1497564332 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc5e406e350 0x7fc5e4070810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec0096c0 tx=0x7fc5ec000bc0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:32.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.152+0000 7fc5f1ffb640 1 -- 172.21.15.114:0/1497564332 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fc5f801f020 con 0x7fc5fc100880 2024-09-20T19:42:32.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:32.342+0000 7fc602d7e640 1 -- 172.21.15.114:0/1497564332 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) -- 0x7fc5fc06aac0 con 0x7fc5e406e350 2024-09-20T19:42:32.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:32 smithi114 ceph-mon[28260]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:32.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:32 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-09-20T19:42:33.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:32 smithi160 ceph-mon[35859]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:33.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:32 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-09-20T19:42:33.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:33.661+0000 7fc5f1ffb640 1 -- 172.21.15.114:0/1497564332 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc5fc06aac0 con 0x7fc5e406e350 2024-09-20T19:42:33.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:33.665+0000 7fc602d7e640 1 -- 172.21.15.114:0/1497564332 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc5e406e350 msgr2=0x7fc5e4070810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:33.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:33.665+0000 7fc602d7e640 1 --2- 172.21.15.114:0/1497564332 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc5e406e350 0x7fc5e4070810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec0096c0 tx=0x7fc5ec000bc0 comp rx=0 tx=0).stop 2024-09-20T19:42:33.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:33.665+0000 7fc602d7e640 1 -- 172.21.15.114:0/1497564332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5fc100880 msgr2=0x7fc5fc197580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:33.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:33.665+0000 7fc602d7e640 1 --2- 172.21.15.114:0/1497564332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5fc100880 0x7fc5fc197580 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc5f800b3d0 tx=0x7fc5f800b890 comp rx=0 tx=0).stop 2024-09-20T19:42:33.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:33.665+0000 7fc602d7e640 1 -- 172.21.15.114:0/1497564332 shutdown_connections 2024-09-20T19:42:33.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:33.665+0000 7fc602d7e640 1 --2- 172.21.15.114:0/1497564332 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc5e406e350 0x7fc5e4070810 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:33.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:33.665+0000 7fc602d7e640 1 --2- 172.21.15.114:0/1497564332 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc5fc101a80 0x7fc5fc197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:33.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:33.665+0000 7fc602d7e640 1 --2- 172.21.15.114:0/1497564332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5fc100880 0x7fc5fc197580 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:33.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:33.665+0000 7fc602d7e640 1 -- 172.21.15.114:0/1497564332 >> 172.21.15.114:0/1497564332 conn(0x7fc5fc0fc010 msgr2=0x7fc5fc06a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:33.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:33.666+0000 7fc602d7e640 1 -- 172.21.15.114:0/1497564332 shutdown_connections 2024-09-20T19:42:33.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:33.667+0000 7fc602d7e640 1 -- 172.21.15.114:0/1497564332 wait complete. 2024-09-20T19:42:33.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:33 smithi114 ceph-mon[28260]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:33.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:33 smithi114 ceph-mon[28260]: from='client.14542 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:33.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:33 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-09-20T19:42:33.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:33 smithi114 ceph-mon[28260]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T19:42:33.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:33 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-09-20T19:42:33.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:42:33.641+0000 7f98a9c0f640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-20T19:42:34.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:33 smithi160 ceph-mon[35859]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:34.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:33 smithi160 ceph-mon[35859]: from='client.14542 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:42:34.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:33 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-09-20T19:42:34.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:33 smithi160 ceph-mon[35859]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T19:42:34.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:33 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-09-20T19:42:34.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- bash -c 'ceph fs dump' 2024-09-20T19:42:34.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-09-20T19:42:34.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T19:42:34.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-09-20T19:42:34.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-20T19:42:34.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-20T19:42:34.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-09-20T19:42:34.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: osdmap e44: 6 total, 6 up, 6 in 2024-09-20T19:42:34.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: fsmap cephfs:0 2024-09-20T19:42:34.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: Saving service mds.cephfs spec with placement count:4 2024-09-20T19:42:34.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:34.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:42:34.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:42:34.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:42:34.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:34.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.scsmgq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T19:42:34.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.scsmgq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-20T19:42:34.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:42:34.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: Deploying daemon mds.cephfs.smithi114.scsmgq on smithi114 2024-09-20T19:42:34.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:34 smithi114 ceph-mon[28260]: pgmap v171: 65 pgs: 8 creating+peering, 51 unknown, 6 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:35.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-09-20T19:42:35.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T19:42:35.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-09-20T19:42:35.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-20T19:42:35.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-20T19:42:35.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-09-20T19:42:35.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: osdmap e44: 6 total, 6 up, 6 in 2024-09-20T19:42:35.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: fsmap cephfs:0 2024-09-20T19:42:35.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: Saving service mds.cephfs spec with placement count:4 2024-09-20T19:42:35.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:35.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:42:35.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:42:35.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:42:35.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:35.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.scsmgq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T19:42:35.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.scsmgq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-20T19:42:35.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:42:35.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: Deploying daemon mds.cephfs.smithi114.scsmgq on smithi114 2024-09-20T19:42:35.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:34 smithi160 ceph-mon[35859]: pgmap v171: 65 pgs: 8 creating+peering, 51 unknown, 6 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:35.348 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:35.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:35 smithi160 ceph-mon[35859]: osdmap e45: 6 total, 6 up, 6 in 2024-09-20T19:42:35.892 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:35 smithi114 ceph-mon[28260]: osdmap e45: 6 total, 6 up, 6 in 2024-09-20T19:42:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.972+0000 7f9e7008c640 1 -- 172.21.15.114:0/3029477719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e68101a80 msgr2=0x7f9e68101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.972+0000 7f9e7008c640 1 --2- 172.21.15.114:0/3029477719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e68101a80 0x7f9e68101f00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009880 tx=0x7f9e6402a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.972+0000 7f9e7008c640 1 -- 172.21.15.114:0/3029477719 shutdown_connections 2024-09-20T19:42:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.972+0000 7f9e7008c640 1 --2- 172.21.15.114:0/3029477719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e68101a80 0x7f9e68101f00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.972+0000 7f9e7008c640 1 --2- 172.21.15.114:0/3029477719 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e68100880 0x7f9e68100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.972+0000 7f9e7008c640 1 -- 172.21.15.114:0/3029477719 >> 172.21.15.114:0/3029477719 conn(0x7f9e680fc010 msgr2=0x7f9e680fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.972+0000 7f9e7008c640 1 -- 172.21.15.114:0/3029477719 shutdown_connections 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.973+0000 7f9e7008c640 1 -- 172.21.15.114:0/3029477719 wait complete. 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.973+0000 7f9e7008c640 1 Processor -- start 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.973+0000 7f9e7008c640 1 -- start start 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.973+0000 7f9e7008c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e68100880 0x7f9e681047a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.973+0000 7f9e7008c640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e68101a80 0x7f9e68102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.973+0000 7f9e7008c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9e681096a0 con 0x7f9e68100880 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.973+0000 7f9e7008c640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f9e68109520 con 0x7f9e68101a80 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.973+0000 7f9e6de28640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e68100880 0x7f9e681047a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.973+0000 7f9e6de28640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e68100880 0x7f9e681047a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53828/0 (socket says 172.21.15.114:53828) 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.974+0000 7f9e6de28640 1 -- 172.21.15.114:0/1632394720 learned_addr learned my addr 172.21.15.114:0/1632394720 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.974+0000 7f9e6de28640 1 -- 172.21.15.114:0/1632394720 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e68101a80 msgr2=0x7f9e68102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.974+0000 7f9e6de28640 1 --2- 172.21.15.114:0/1632394720 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e68101a80 0x7f9e68102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.974+0000 7f9e6de28640 1 -- 172.21.15.114:0/1632394720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e681033a0 con 0x7f9e68100880 2024-09-20T19:42:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.974+0000 7f9e6de28640 1 --2- 172.21.15.114:0/1632394720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e68100880 0x7f9e681047a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800c8f0 tx=0x7f9e5800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:35.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.974+0000 7f9e5effd640 1 -- 172.21.15.114:0/1632394720 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f9e580079b0 con 0x7f9e68100880 2024-09-20T19:42:35.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.974+0000 7f9e5effd640 1 -- 172.21.15.114:0/1632394720 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9e58007b50 con 0x7f9e68100880 2024-09-20T19:42:35.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.974+0000 7f9e7008c640 1 -- 172.21.15.114:0/1632394720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e68103690 con 0x7f9e68100880 2024-09-20T19:42:35.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.974+0000 7f9e5effd640 1 -- 172.21.15.114:0/1632394720 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f9e58018610 con 0x7f9e68100880 2024-09-20T19:42:35.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.974+0000 7f9e7008c640 1 -- 172.21.15.114:0/1632394720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e68103c20 con 0x7f9e68100880 2024-09-20T19:42:35.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.975+0000 7f9e7008c640 1 -- 172.21.15.114:0/1632394720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e68100c80 con 0x7f9e68100880 2024-09-20T19:42:35.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.975+0000 7f9e5effd640 1 -- 172.21.15.114:0/1632394720 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f9e58007d00 con 0x7f9e68100880 2024-09-20T19:42:35.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.976+0000 7f9e5effd640 1 --2- 172.21.15.114:0/1632394720 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f9e5406e2b0 0x7f9e54070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:35.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.976+0000 7f9e5effd640 1 -- 172.21.15.114:0/1632394720 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f9e5808e3c0 con 0x7f9e68100880 2024-09-20T19:42:35.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.976+0000 7f9e6d627640 1 --2- 172.21.15.114:0/1632394720 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f9e5406e2b0 0x7f9e54070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:35.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.976+0000 7f9e6d627640 1 --2- 172.21.15.114:0/1632394720 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f9e5406e2b0 0x7f9e54070770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9e6402a9c0 tx=0x7f9e64000bc0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:35.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:35.980+0000 7f9e5effd640 1 -- 172.21.15.114:0/1632394720 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f9e58014030 con 0x7f9e68100880 2024-09-20T19:42:36.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:36.191+0000 7f9e7008c640 1 -- 172.21.15.114:0/1632394720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f9e6819f740 con 0x7f9e68100880 2024-09-20T19:42:36.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:36.192+0000 7f9e5effd640 1 -- 172.21.15.114:0/1632394720 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) ==== 75+0+1272 (secure 0 0 0) 0x7f9e5805bd20 con 0x7f9e68100880 2024-09-20T19:42:36.194 INFO:teuthology.orchestra.run.smithi114.stdout:e2 2024-09-20T19:42:36.194 INFO:teuthology.orchestra.run.smithi114.stdout:btime 2024-09-20T19:42:33:642328+0000 2024-09-20T19:42:36.194 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T19:42:36.194 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-20T19:42:36.194 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-09-20T19:42:36.194 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:42:36.194 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-09-20T19:42:36.194 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-09-20T19:42:36.194 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 2 2024-09-20T19:42:36.194 INFO:teuthology.orchestra.run.smithi114.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-20T19:42:36.194 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-09-20T19:42:33.642256+0000 2024-09-20T19:42:36.194 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-09-20T19:42:33.642256+0000 2024-09-20T19:42:36.195 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-09-20T19:42:36.195 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-09-20T19:42:36.195 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-09-20T19:42:36.195 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-09-20T19:42:36.195 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-09-20T19:42:36.195 INFO:teuthology.orchestra.run.smithi114.stdout:max_xattr_size 65536 2024-09-20T19:42:36.195 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-09-20T19:42:36.195 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-09-20T19:42:36.195 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 0 2024-09-20T19:42:36.195 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-20T19:42:36.195 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 1 2024-09-20T19:42:36.196 INFO:teuthology.orchestra.run.smithi114.stdout:in 2024-09-20T19:42:36.196 INFO:teuthology.orchestra.run.smithi114.stdout:up {} 2024-09-20T19:42:36.196 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-09-20T19:42:36.196 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-09-20T19:42:36.196 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-09-20T19:42:36.196 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-09-20T19:42:36.196 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-09-20T19:42:36.196 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data disabled 2024-09-20T19:42:36.196 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-09-20T19:42:36.196 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-09-20T19:42:36.196 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 0 2024-09-20T19:42:36.196 INFO:teuthology.orchestra.run.smithi114.stdout:qdb_cluster leader: 0 members: 2024-09-20T19:42:36.196 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:42:36.196 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:42:36.197 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 2 2024-09-20T19:42:36.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:36.194+0000 7f9e7008c640 1 -- 172.21.15.114:0/1632394720 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f9e5406e2b0 msgr2=0x7f9e54070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:36.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:36.194+0000 7f9e7008c640 1 --2- 172.21.15.114:0/1632394720 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f9e5406e2b0 0x7f9e54070770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9e6402a9c0 tx=0x7f9e64000bc0 comp rx=0 tx=0).stop 2024-09-20T19:42:36.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:36.194+0000 7f9e7008c640 1 -- 172.21.15.114:0/1632394720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e68100880 msgr2=0x7f9e681047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:36.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:36.194+0000 7f9e7008c640 1 --2- 172.21.15.114:0/1632394720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e68100880 0x7f9e681047a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800c8f0 tx=0x7f9e5800cdb0 comp rx=0 tx=0).stop 2024-09-20T19:42:36.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:36.195+0000 7f9e7008c640 1 -- 172.21.15.114:0/1632394720 shutdown_connections 2024-09-20T19:42:36.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:36.195+0000 7f9e7008c640 1 --2- 172.21.15.114:0/1632394720 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f9e5406e2b0 0x7f9e54070770 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:36.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:36.195+0000 7f9e7008c640 1 --2- 172.21.15.114:0/1632394720 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9e68101a80 0x7f9e68102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:36.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:36.195+0000 7f9e7008c640 1 --2- 172.21.15.114:0/1632394720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e68100880 0x7f9e681047a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:36.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:36.195+0000 7f9e7008c640 1 -- 172.21.15.114:0/1632394720 >> 172.21.15.114:0/1632394720 conn(0x7f9e680fc010 msgr2=0x7f9e6806a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:36.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:36.195+0000 7f9e7008c640 1 -- 172.21.15.114:0/1632394720 shutdown_connections 2024-09-20T19:42:36.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:36.196+0000 7f9e7008c640 1 -- 172.21.15.114:0/1632394720 wait complete. 2024-09-20T19:42:36.942 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:36 smithi114 ceph-mon[28260]: osdmap e46: 6 total, 6 up, 6 in 2024-09-20T19:42:36.942 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:36 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1632394720' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-20T19:42:36.942 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:36 smithi114 ceph-mon[28260]: pgmap v174: 65 pgs: 20 creating+peering, 27 unknown, 18 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:37.021 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T19:42:37.030 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-09-20T19:42:37.031 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-09-20T19:42:37.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:36 smithi160 ceph-mon[35859]: osdmap e46: 6 total, 6 up, 6 in 2024-09-20T19:42:37.032 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:36 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1632394720' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-20T19:42:37.032 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:36 smithi160 ceph-mon[35859]: pgmap v174: 65 pgs: 20 creating+peering, 27 unknown, 18 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:37.369 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:37.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.814+0000 7f8a54c81640 1 -- 172.21.15.114:0/1039740049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a50101e10 msgr2=0x7f8a50102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:37.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.814+0000 7f8a54c81640 1 --2- 172.21.15.114:0/1039740049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a50101e10 0x7f8a50102210 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8a40009880 tx=0x7f8a4002a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:37.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.814+0000 7f8a54c81640 1 -- 172.21.15.114:0/1039740049 shutdown_connections 2024-09-20T19:42:37.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.814+0000 7f8a54c81640 1 --2- 172.21.15.114:0/1039740049 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a50100b50 0x7f8a50100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:37.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.814+0000 7f8a54c81640 1 --2- 172.21.15.114:0/1039740049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a50101e10 0x7f8a50102210 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:37.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.814+0000 7f8a54c81640 1 -- 172.21.15.114:0/1039740049 >> 172.21.15.114:0/1039740049 conn(0x7f8a500fb7b0 msgr2=0x7f8a500fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:37.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.814+0000 7f8a54c81640 1 -- 172.21.15.114:0/1039740049 shutdown_connections 2024-09-20T19:42:37.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.814+0000 7f8a54c81640 1 -- 172.21.15.114:0/1039740049 wait complete. 2024-09-20T19:42:37.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.815+0000 7f8a54c81640 1 Processor -- start 2024-09-20T19:42:37.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.815+0000 7f8a54c81640 1 -- start start 2024-09-20T19:42:37.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.815+0000 7f8a54c81640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a50100b50 0x7f8a50197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:37.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.815+0000 7f8a54c81640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a50101e10 0x7f8a50197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:37.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.815+0000 7f8a54c81640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8a501096c0 con 0x7f8a50100b50 2024-09-20T19:42:37.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.815+0000 7f8a54c81640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f8a50109540 con 0x7f8a50101e10 2024-09-20T19:42:37.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.816+0000 7f8a4e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a50100b50 0x7f8a50197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:37.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.816+0000 7f8a4e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a50100b50 0x7f8a50197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53864/0 (socket says 172.21.15.114:53864) 2024-09-20T19:42:37.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.816+0000 7f8a4e59c640 1 -- 172.21.15.114:0/2603853974 learned_addr learned my addr 172.21.15.114:0/2603853974 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:37.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.816+0000 7f8a4dd9b640 1 --2- 172.21.15.114:0/2603853974 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a50101e10 0x7f8a50197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:37.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.816+0000 7f8a4e59c640 1 -- 172.21.15.114:0/2603853974 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a50101e10 msgr2=0x7f8a50197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:37.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.816+0000 7f8a4e59c640 1 --2- 172.21.15.114:0/2603853974 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a50101e10 0x7f8a50197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:37.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.816+0000 7f8a4e59c640 1 -- 172.21.15.114:0/2603853974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a501980f0 con 0x7f8a50100b50 2024-09-20T19:42:37.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.816+0000 7f8a4e59c640 1 --2- 172.21.15.114:0/2603853974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a50100b50 0x7f8a50197530 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8a40006010 tx=0x7f8a400045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:37.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.816+0000 7f8a3f7fe640 1 -- 172.21.15.114:0/2603853974 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8a4003f070 con 0x7f8a50100b50 2024-09-20T19:42:37.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.816+0000 7f8a3f7fe640 1 -- 172.21.15.114:0/2603853974 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8a40004a50 con 0x7f8a50100b50 2024-09-20T19:42:37.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.816+0000 7f8a54c81640 1 -- 172.21.15.114:0/2603853974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a5019cad0 con 0x7f8a50100b50 2024-09-20T19:42:37.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.816+0000 7f8a3f7fe640 1 -- 172.21.15.114:0/2603853974 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8a40035030 con 0x7f8a50100b50 2024-09-20T19:42:37.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.816+0000 7f8a54c81640 1 -- 172.21.15.114:0/2603853974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a5019d060 con 0x7f8a50100b50 2024-09-20T19:42:37.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.817+0000 7f8a54c81640 1 -- 172.21.15.114:0/2603853974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a50100fd0 con 0x7f8a50100b50 2024-09-20T19:42:37.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.818+0000 7f8a3f7fe640 1 -- 172.21.15.114:0/2603853974 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f8a40042070 con 0x7f8a50100b50 2024-09-20T19:42:37.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.818+0000 7f8a3f7fe640 1 --2- 172.21.15.114:0/2603853974 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8a3806e2b0 0x7f8a38070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:37.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.818+0000 7f8a3f7fe640 1 -- 172.21.15.114:0/2603853974 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f8a400aed20 con 0x7f8a50100b50 2024-09-20T19:42:37.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.818+0000 7f8a4dd9b640 1 --2- 172.21.15.114:0/2603853974 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8a3806e2b0 0x7f8a38070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:37.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.819+0000 7f8a4dd9b640 1 --2- 172.21.15.114:0/2603853974 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8a3806e2b0 0x7f8a38070770 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8a44005fd0 tx=0x7f8a44005850 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:37.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:37.823+0000 7f8a3f7fe640 1 -- 172.21.15.114:0/2603853974 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f8a40004c00 con 0x7f8a50100b50 2024-09-20T19:42:38.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:38.025+0000 7f8a54c81640 1 -- 172.21.15.114:0/2603853974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) -- 0x7f8a5019d290 con 0x7f8a50100b50 2024-09-20T19:42:38.473 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:38.473 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:38.473 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:38.473 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi160.xirihp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T19:42:38.473 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi160.xirihp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-20T19:42:38.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:42:38.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: Deploying daemon mds.cephfs.smithi160.xirihp on smithi160 2024-09-20T19:42:38.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: mds.? [v2:172.21.15.114:6826/500124176,v1:172.21.15.114:6827/500124176] up:boot 2024-09-20T19:42:38.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: daemon mds.cephfs.smithi114.scsmgq assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-09-20T19:42:38.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-20T19:42:38.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-20T19:42:38.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: Cluster is now healthy 2024-09-20T19:42:38.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: fsmap cephfs:0 1 up:standby 2024-09-20T19:42:38.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.scsmgq"}]: dispatch 2024-09-20T19:42:38.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:creating} 2024-09-20T19:42:38.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: daemon mds.cephfs.smithi114.scsmgq is now active in filesystem cephfs as rank 0 2024-09-20T19:42:38.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:38 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/2603853974' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-09-20T19:42:38.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:38.668+0000 7f8a3f7fe640 1 -- 172.21.15.114:0/2603853974 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v5) ==== 105+0+0 (secure 0 0 0) 0x7f8a4007c680 con 0x7f8a50100b50 2024-09-20T19:42:38.670 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:38.670 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:38.670 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:38.670 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi160.xirihp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T19:42:38.671 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi160.xirihp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-20T19:42:38.671 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:42:38.671 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: Deploying daemon mds.cephfs.smithi160.xirihp on smithi160 2024-09-20T19:42:38.671 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: mds.? [v2:172.21.15.114:6826/500124176,v1:172.21.15.114:6827/500124176] up:boot 2024-09-20T19:42:38.671 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: daemon mds.cephfs.smithi114.scsmgq assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-09-20T19:42:38.671 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-20T19:42:38.671 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-20T19:42:38.671 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: Cluster is now healthy 2024-09-20T19:42:38.671 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: fsmap cephfs:0 1 up:standby 2024-09-20T19:42:38.671 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.scsmgq"}]: dispatch 2024-09-20T19:42:38.671 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:creating} 2024-09-20T19:42:38.671 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: daemon mds.cephfs.smithi114.scsmgq is now active in filesystem cephfs as rank 0 2024-09-20T19:42:38.671 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:38 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/2603853974' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-09-20T19:42:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:38.671+0000 7f8a54c81640 1 -- 172.21.15.114:0/2603853974 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8a3806e2b0 msgr2=0x7f8a38070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:38.671+0000 7f8a54c81640 1 --2- 172.21.15.114:0/2603853974 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8a3806e2b0 0x7f8a38070770 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8a44005fd0 tx=0x7f8a44005850 comp rx=0 tx=0).stop 2024-09-20T19:42:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:38.671+0000 7f8a54c81640 1 -- 172.21.15.114:0/2603853974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a50100b50 msgr2=0x7f8a50197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:38.671+0000 7f8a54c81640 1 --2- 172.21.15.114:0/2603853974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a50100b50 0x7f8a50197530 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8a40006010 tx=0x7f8a400045e0 comp rx=0 tx=0).stop 2024-09-20T19:42:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:38.671+0000 7f8a54c81640 1 -- 172.21.15.114:0/2603853974 shutdown_connections 2024-09-20T19:42:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:38.671+0000 7f8a54c81640 1 --2- 172.21.15.114:0/2603853974 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f8a3806e2b0 0x7f8a38070770 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:38.671+0000 7f8a54c81640 1 --2- 172.21.15.114:0/2603853974 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8a50101e10 0x7f8a50197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:38.671+0000 7f8a54c81640 1 --2- 172.21.15.114:0/2603853974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a50100b50 0x7f8a50197530 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:38.671+0000 7f8a54c81640 1 -- 172.21.15.114:0/2603853974 >> 172.21.15.114:0/2603853974 conn(0x7f8a500fb7b0 msgr2=0x7f8a500ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:38.671+0000 7f8a54c81640 1 -- 172.21.15.114:0/2603853974 shutdown_connections 2024-09-20T19:42:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:38.672+0000 7f8a54c81640 1 -- 172.21.15.114:0/2603853974 wait complete. 2024-09-20T19:42:39.364 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T19:42:39.374 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-09-20T19:42:39.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-09-20T19:42:39.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:39 smithi160 ceph-mon[35859]: pgmap v175: 65 pgs: 20 creating+peering, 45 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:39.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:39 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/2603853974' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-09-20T19:42:39.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:39 smithi160 ceph-mon[35859]: mds.? [v2:172.21.15.114:6826/500124176,v1:172.21.15.114:6827/500124176] up:active 2024-09-20T19:42:39.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:39 smithi160 ceph-mon[35859]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 2024-09-20T19:42:39.627 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:39.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:39 smithi114 ceph-mon[28260]: pgmap v175: 65 pgs: 20 creating+peering, 45 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:39.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:39 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/2603853974' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-09-20T19:42:39.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:39 smithi114 ceph-mon[28260]: mds.? [v2:172.21.15.114:6826/500124176,v1:172.21.15.114:6827/500124176] up:active 2024-09-20T19:42:39.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:39 smithi114 ceph-mon[28260]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 2024-09-20T19:42:40.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.119+0000 7f304af7e640 1 -- 172.21.15.114:0/1364281740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3044102b50 msgr2=0x7f3044104fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:40.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.119+0000 7f304af7e640 1 --2- 172.21.15.114:0/1364281740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3044102b50 0x7f3044104fc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3030009880 tx=0x7f303002a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:40.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.119+0000 7f304af7e640 1 -- 172.21.15.114:0/1364281740 shutdown_connections 2024-09-20T19:42:40.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.119+0000 7f304af7e640 1 --2- 172.21.15.114:0/1364281740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3044102b50 0x7f3044104fc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:40.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.119+0000 7f304af7e640 1 --2- 172.21.15.114:0/1364281740 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3044100200 0x7f3044102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:40.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.119+0000 7f304af7e640 1 -- 172.21.15.114:0/1364281740 >> 172.21.15.114:0/1364281740 conn(0x7f30440fc010 msgr2=0x7f30440fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:40.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.119+0000 7f304af7e640 1 -- 172.21.15.114:0/1364281740 shutdown_connections 2024-09-20T19:42:40.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.119+0000 7f304af7e640 1 -- 172.21.15.114:0/1364281740 wait complete. 2024-09-20T19:42:40.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.120+0000 7f304af7e640 1 Processor -- start 2024-09-20T19:42:40.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.120+0000 7f304af7e640 1 -- start start 2024-09-20T19:42:40.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.120+0000 7f304af7e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3044100200 0x7f304419da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:40.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.120+0000 7f304af7e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3044102b50 0x7f304419dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:40.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.120+0000 7f304af7e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3044110db0 con 0x7f3044102b50 2024-09-20T19:42:40.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.120+0000 7f304af7e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f3044110c30 con 0x7f3044100200 2024-09-20T19:42:40.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.120+0000 7f3043fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3044102b50 0x7f304419dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:40.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.120+0000 7f3048d1a640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3044100200 0x7f304419da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:40.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.120+0000 7f3043fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3044102b50 0x7f304419dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53882/0 (socket says 172.21.15.114:53882) 2024-09-20T19:42:40.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.120+0000 7f3048d1a640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3044100200 0x7f304419da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.114:37884/0 (socket says 172.21.15.114:37884) 2024-09-20T19:42:40.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.120+0000 7f3043fff640 1 -- 172.21.15.114:0/3748847683 learned_addr learned my addr 172.21.15.114:0/3748847683 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:40.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.120+0000 7f3043fff640 1 -- 172.21.15.114:0/3748847683 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3044100200 msgr2=0x7f304419da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:40.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.120+0000 7f3043fff640 1 --2- 172.21.15.114:0/3748847683 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3044100200 0x7f304419da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:40.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.120+0000 7f3043fff640 1 -- 172.21.15.114:0/3748847683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f304419e630 con 0x7f3044102b50 2024-09-20T19:42:40.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.121+0000 7f3043fff640 1 --2- 172.21.15.114:0/3748847683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3044102b50 0x7f304419dfb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3030006010 tx=0x7f30300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:40.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.121+0000 7f3041ffb640 1 -- 172.21.15.114:0/3748847683 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f303003f070 con 0x7f3044102b50 2024-09-20T19:42:40.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.121+0000 7f304af7e640 1 -- 172.21.15.114:0/3748847683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30441a3010 con 0x7f3044102b50 2024-09-20T19:42:40.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.121+0000 7f3041ffb640 1 -- 172.21.15.114:0/3748847683 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3030004ac0 con 0x7f3044102b50 2024-09-20T19:42:40.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.121+0000 7f3041ffb640 1 -- 172.21.15.114:0/3748847683 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3030035030 con 0x7f3044102b50 2024-09-20T19:42:40.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.121+0000 7f304af7e640 1 -- 172.21.15.114:0/3748847683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30441a35a0 con 0x7f3044102b50 2024-09-20T19:42:40.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.122+0000 7f304af7e640 1 -- 172.21.15.114:0/3748847683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3044100600 con 0x7f3044102b50 2024-09-20T19:42:40.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.122+0000 7f3041ffb640 1 -- 172.21.15.114:0/3748847683 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f3030042070 con 0x7f3044102b50 2024-09-20T19:42:40.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.123+0000 7f3041ffb640 1 --2- 172.21.15.114:0/3748847683 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f303406e2b0 0x7f3034070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:40.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.123+0000 7f3041ffb640 1 -- 172.21.15.114:0/3748847683 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f30300aed20 con 0x7f3044102b50 2024-09-20T19:42:40.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.123+0000 7f3048d1a640 1 --2- 172.21.15.114:0/3748847683 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f303406e2b0 0x7f3034070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:40.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.123+0000 7f3048d1a640 1 --2- 172.21.15.114:0/3748847683 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f303406e2b0 0x7f3034070770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3038005fd0 tx=0x7f303800b390 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:40.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.127+0000 7f3041ffb640 1 -- 172.21.15.114:0/3748847683 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f3030080330 con 0x7f3044102b50 2024-09-20T19:42:40.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:40.347+0000 7f304af7e640 1 -- 172.21.15.114:0/3748847683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f30441a3910 con 0x7f3044102b50 2024-09-20T19:42:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:40 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:40 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:40 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:40 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.tlrcnf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T19:42:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:40 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.tlrcnf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-20T19:42:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:40 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:42:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:40 smithi114 ceph-mon[28260]: Deploying daemon mds.cephfs.smithi114.tlrcnf on smithi114 2024-09-20T19:42:41.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:40 smithi114 ceph-mon[28260]: pgmap v176: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s wr, 3 op/s 2024-09-20T19:42:41.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:40 smithi114 ceph-mon[28260]: mds.? [v2:172.21.15.160:6824/2564850174,v1:172.21.15.160:6825/2564850174] up:boot 2024-09-20T19:42:41.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:40 smithi114 ceph-mon[28260]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 1 up:standby 2024-09-20T19:42:41.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:40 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi160.xirihp"}]: dispatch 2024-09-20T19:42:41.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:40 smithi114 ceph-mon[28260]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 1 up:standby 2024-09-20T19:42:41.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:40 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3748847683' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-09-20T19:42:41.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:40 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:41.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:40 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:41.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:40 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:41.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:40 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.tlrcnf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T19:42:41.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:40 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.tlrcnf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-20T19:42:41.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:40 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:42:41.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:40 smithi160 ceph-mon[35859]: Deploying daemon mds.cephfs.smithi114.tlrcnf on smithi114 2024-09-20T19:42:41.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:40 smithi160 ceph-mon[35859]: pgmap v176: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s wr, 3 op/s 2024-09-20T19:42:41.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:40 smithi160 ceph-mon[35859]: mds.? [v2:172.21.15.160:6824/2564850174,v1:172.21.15.160:6825/2564850174] up:boot 2024-09-20T19:42:41.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:40 smithi160 ceph-mon[35859]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 1 up:standby 2024-09-20T19:42:41.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:40 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi160.xirihp"}]: dispatch 2024-09-20T19:42:41.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:40 smithi160 ceph-mon[35859]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 1 up:standby 2024-09-20T19:42:41.280 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:40 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/3748847683' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-09-20T19:42:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:41.340+0000 7f3041ffb640 1 -- 172.21.15.114:0/3748847683 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7f3030084b60 con 0x7f3044102b50 2024-09-20T19:42:41.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:41.343+0000 7f304af7e640 1 -- 172.21.15.114:0/3748847683 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f303406e2b0 msgr2=0x7f3034070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:41.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:41.343+0000 7f304af7e640 1 --2- 172.21.15.114:0/3748847683 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f303406e2b0 0x7f3034070770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3038005fd0 tx=0x7f303800b390 comp rx=0 tx=0).stop 2024-09-20T19:42:41.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:41.344+0000 7f304af7e640 1 -- 172.21.15.114:0/3748847683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3044102b50 msgr2=0x7f304419dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:41.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:41.344+0000 7f304af7e640 1 --2- 172.21.15.114:0/3748847683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3044102b50 0x7f304419dfb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3030006010 tx=0x7f30300046c0 comp rx=0 tx=0).stop 2024-09-20T19:42:41.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:41.344+0000 7f304af7e640 1 -- 172.21.15.114:0/3748847683 shutdown_connections 2024-09-20T19:42:41.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:41.344+0000 7f304af7e640 1 --2- 172.21.15.114:0/3748847683 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f303406e2b0 0x7f3034070770 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:41.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:41.344+0000 7f304af7e640 1 --2- 172.21.15.114:0/3748847683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3044102b50 0x7f304419dfb0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:41.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:41.344+0000 7f304af7e640 1 --2- 172.21.15.114:0/3748847683 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3044100200 0x7f304419da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:41.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:41.344+0000 7f304af7e640 1 -- 172.21.15.114:0/3748847683 >> 172.21.15.114:0/3748847683 conn(0x7f30440fc010 msgr2=0x7f30440fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:41.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:41.344+0000 7f304af7e640 1 -- 172.21.15.114:0/3748847683 shutdown_connections 2024-09-20T19:42:41.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:41.344+0000 7f304af7e640 1 -- 172.21.15.114:0/3748847683 wait complete. 2024-09-20T19:42:42.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:41 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:42:42.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:41 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3748847683' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-09-20T19:42:42.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:41 smithi114 ceph-mon[28260]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 1 up:standby 2024-09-20T19:42:42.071 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T19:42:42.088 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-09-20T19:42:42.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-09-20T19:42:42.123 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:41 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:42:42.123 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:41 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/3748847683' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-09-20T19:42:42.123 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:41 smithi160 ceph-mon[35859]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 1 up:standby 2024-09-20T19:42:42.644 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.079+0000 7f6659df9640 1 -- 172.21.15.114:0/163176843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654100880 msgr2=0x7f6654100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.079+0000 7f6659df9640 1 --2- 172.21.15.114:0/163176843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654100880 0x7f6654100c80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6644009880 tx=0x7f664402a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.079+0000 7f6659df9640 1 -- 172.21.15.114:0/163176843 shutdown_connections 2024-09-20T19:42:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.079+0000 7f6659df9640 1 --2- 172.21.15.114:0/163176843 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6654101a80 0x7f6654101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.079+0000 7f6659df9640 1 --2- 172.21.15.114:0/163176843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654100880 0x7f6654100c80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.079+0000 7f6659df9640 1 -- 172.21.15.114:0/163176843 >> 172.21.15.114:0/163176843 conn(0x7f66540fc010 msgr2=0x7f66540fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.079+0000 7f6659df9640 1 -- 172.21.15.114:0/163176843 shutdown_connections 2024-09-20T19:42:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.079+0000 7f6659df9640 1 -- 172.21.15.114:0/163176843 wait complete. 2024-09-20T19:42:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.080+0000 7f6659df9640 1 Processor -- start 2024-09-20T19:42:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.080+0000 7f6659df9640 1 -- start start 2024-09-20T19:42:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.080+0000 7f6659df9640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6654100880 0x7f66541047a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.080+0000 7f6659df9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654101a80 0x7f6654102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.080+0000 7f6659df9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6654109720 con 0x7f6654101a80 2024-09-20T19:42:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.080+0000 7f6659df9640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f66541095a0 con 0x7f6654100880 2024-09-20T19:42:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.080+0000 7f6652ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654101a80 0x7f6654102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.080+0000 7f6652ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654101a80 0x7f6654102e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53932/0 (socket says 172.21.15.114:53932) 2024-09-20T19:42:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.080+0000 7f6652ffd640 1 -- 172.21.15.114:0/1017327713 learned_addr learned my addr 172.21.15.114:0/1017327713 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.080+0000 7f66537fe640 1 --2- 172.21.15.114:0/1017327713 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6654100880 0x7f66541047a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.080+0000 7f6652ffd640 1 -- 172.21.15.114:0/1017327713 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6654100880 msgr2=0x7f66541047a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.080+0000 7f6652ffd640 1 --2- 172.21.15.114:0/1017327713 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6654100880 0x7f66541047a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.080+0000 7f6652ffd640 1 -- 172.21.15.114:0/1017327713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66541033a0 con 0x7f6654101a80 2024-09-20T19:42:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.081+0000 7f6652ffd640 1 --2- 172.21.15.114:0/1017327713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654101a80 0x7f6654102e60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f664800b420 tx=0x7f664800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.081+0000 7f6650ff9640 1 -- 172.21.15.114:0/1017327713 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6648011630 con 0x7f6654101a80 2024-09-20T19:42:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.081+0000 7f6659df9640 1 -- 172.21.15.114:0/1017327713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6654103690 con 0x7f6654101a80 2024-09-20T19:42:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.081+0000 7f6650ff9640 1 -- 172.21.15.114:0/1017327713 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6648016070 con 0x7f6654101a80 2024-09-20T19:42:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.081+0000 7f6650ff9640 1 -- 172.21.15.114:0/1017327713 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6648010d40 con 0x7f6654101a80 2024-09-20T19:42:43.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.081+0000 7f6659df9640 1 -- 172.21.15.114:0/1017327713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6654103ca0 con 0x7f6654101a80 2024-09-20T19:42:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.082+0000 7f6659df9640 1 -- 172.21.15.114:0/1017327713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6654100c80 con 0x7f6654101a80 2024-09-20T19:42:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.083+0000 7f6650ff9640 1 -- 172.21.15.114:0/1017327713 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f664801b460 con 0x7f6654101a80 2024-09-20T19:42:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.083+0000 7f6650ff9640 1 --2- 172.21.15.114:0/1017327713 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f663c06e2b0 0x7f663c070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.083+0000 7f6650ff9640 1 -- 172.21.15.114:0/1017327713 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f664808d970 con 0x7f6654101a80 2024-09-20T19:42:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.083+0000 7f66537fe640 1 --2- 172.21.15.114:0/1017327713 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f663c06e2b0 0x7f663c070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.083+0000 7f66537fe640 1 --2- 172.21.15.114:0/1017327713 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f663c06e2b0 0x7f663c070770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6644009230 tx=0x7f66440058d0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:43.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.087+0000 7f6650ff9640 1 -- 172.21.15.114:0/1017327713 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f664808fe20 con 0x7f6654101a80 2024-09-20T19:42:43.203 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:42 smithi160 ceph-mon[35859]: pgmap v177: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s wr, 4 op/s 2024-09-20T19:42:43.203 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:42 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:43.203 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:42 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:43.203 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:42 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:43.203 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:42 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi160.lylyja", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T19:42:43.203 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:42 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi160.lylyja", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-20T19:42:43.203 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:42 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:42:43.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:42 smithi114 ceph-mon[28260]: pgmap v177: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s wr, 4 op/s 2024-09-20T19:42:43.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:42 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:43.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:42 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:43.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:42 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:43.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:42 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi160.lylyja", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T19:42:43.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:42 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi160.lylyja", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-20T19:42:43.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:42 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:42:43.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.290+0000 7f6659df9640 1 -- 172.21.15.114:0/1017327713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) -- 0x7f665419f6f0 con 0x7f6654101a80 2024-09-20T19:42:43.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.888+0000 7f6650ff9640 1 -- 172.21.15.114:0/1017327713 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v10) ==== 168+0+0 (secure 0 0 0) 0x7f664805b2d0 con 0x7f6654101a80 2024-09-20T19:42:43.890 INFO:teuthology.orchestra.run.smithi114.stderr:inline data enabled 2024-09-20T19:42:43.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.891+0000 7f6659df9640 1 -- 172.21.15.114:0/1017327713 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f663c06e2b0 msgr2=0x7f663c070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:43.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.891+0000 7f6659df9640 1 --2- 172.21.15.114:0/1017327713 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f663c06e2b0 0x7f663c070770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6644009230 tx=0x7f66440058d0 comp rx=0 tx=0).stop 2024-09-20T19:42:43.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.891+0000 7f6659df9640 1 -- 172.21.15.114:0/1017327713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654101a80 msgr2=0x7f6654102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:43.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.891+0000 7f6659df9640 1 --2- 172.21.15.114:0/1017327713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654101a80 0x7f6654102e60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f664800b420 tx=0x7f664800b8e0 comp rx=0 tx=0).stop 2024-09-20T19:42:43.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.891+0000 7f6659df9640 1 -- 172.21.15.114:0/1017327713 shutdown_connections 2024-09-20T19:42:43.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.891+0000 7f6659df9640 1 --2- 172.21.15.114:0/1017327713 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f663c06e2b0 0x7f663c070770 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:43.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.891+0000 7f6659df9640 1 --2- 172.21.15.114:0/1017327713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654101a80 0x7f6654102e60 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:43.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.891+0000 7f6659df9640 1 --2- 172.21.15.114:0/1017327713 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6654100880 0x7f66541047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:43.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.891+0000 7f6659df9640 1 -- 172.21.15.114:0/1017327713 >> 172.21.15.114:0/1017327713 conn(0x7f66540fc010 msgr2=0x7f665406a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:43.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.891+0000 7f6659df9640 1 -- 172.21.15.114:0/1017327713 shutdown_connections 2024-09-20T19:42:43.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:43.891+0000 7f6659df9640 1 -- 172.21.15.114:0/1017327713 wait complete. 2024-09-20T19:42:44.191 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:43 smithi160 ceph-mon[35859]: Deploying daemon mds.cephfs.smithi160.lylyja on smithi160 2024-09-20T19:42:44.191 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:43 smithi160 ceph-mon[35859]: mds.? [v2:172.21.15.114:6828/3069148139,v1:172.21.15.114:6829/3069148139] up:boot 2024-09-20T19:42:44.192 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:43 smithi160 ceph-mon[35859]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 2 up:standby 2024-09-20T19:42:44.192 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:43 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.tlrcnf"}]: dispatch 2024-09-20T19:42:44.192 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:43 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1017327713' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-20T19:42:44.200 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:43 smithi114 ceph-mon[28260]: Deploying daemon mds.cephfs.smithi160.lylyja on smithi160 2024-09-20T19:42:44.200 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:43 smithi114 ceph-mon[28260]: mds.? [v2:172.21.15.114:6828/3069148139,v1:172.21.15.114:6829/3069148139] up:boot 2024-09-20T19:42:44.200 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:43 smithi114 ceph-mon[28260]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 2 up:standby 2024-09-20T19:42:44.200 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:43 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.tlrcnf"}]: dispatch 2024-09-20T19:42:44.200 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:43 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1017327713' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-20T19:42:44.527 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T19:42:44.538 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-09-20T19:42:44.538 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- bash -c 'ceph fs dump' 2024-09-20T19:42:44.790 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:45.195 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:44 smithi114 ceph-mon[28260]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-09-20T19:42:45.196 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:44 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/1017327713' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-09-20T19:42:45.196 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:44 smithi114 ceph-mon[28260]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 2 up:standby 2024-09-20T19:42:45.196 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:44 smithi114 ceph-mon[28260]: pgmap v178: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s wr, 4 op/s 2024-09-20T19:42:45.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.193+0000 7f35f19e5640 1 -- 172.21.15.114:0/1217304895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35ec101a40 msgr2=0x7f35ec101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:45.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.193+0000 7f35f19e5640 1 --2- 172.21.15.114:0/1217304895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35ec101a40 0x7f35ec101ec0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009880 tx=0x7f35e002a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:45.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.193+0000 7f35f19e5640 1 -- 172.21.15.114:0/1217304895 shutdown_connections 2024-09-20T19:42:45.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.193+0000 7f35f19e5640 1 --2- 172.21.15.114:0/1217304895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35ec101a40 0x7f35ec101ec0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:45.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.193+0000 7f35f19e5640 1 --2- 172.21.15.114:0/1217304895 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35ec100840 0x7f35ec100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:45.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.193+0000 7f35f19e5640 1 -- 172.21.15.114:0/1217304895 >> 172.21.15.114:0/1217304895 conn(0x7f35ec0fbff0 msgr2=0x7f35ec0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:45.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.193+0000 7f35f19e5640 1 -- 172.21.15.114:0/1217304895 shutdown_connections 2024-09-20T19:42:45.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.193+0000 7f35f19e5640 1 -- 172.21.15.114:0/1217304895 wait complete. 2024-09-20T19:42:45.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.194+0000 7f35f19e5640 1 Processor -- start 2024-09-20T19:42:45.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.194+0000 7f35f19e5640 1 -- start start 2024-09-20T19:42:45.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35f19e5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35ec100840 0x7f35ec06a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:45.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35f19e5640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35ec101a40 0x7f35ec19c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:45.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35f19e5640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f35ec109680 con 0x7f35ec100840 2024-09-20T19:42:45.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35f19e5640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f35ec109500 con 0x7f35ec101a40 2024-09-20T19:42:45.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35eaffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35ec100840 0x7f35ec06a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:45.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35ea7fc640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35ec101a40 0x7f35ec19c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:45.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35eaffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35ec100840 0x7f35ec06a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53942/0 (socket says 172.21.15.114:53942) 2024-09-20T19:42:45.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35eaffd640 1 -- 172.21.15.114:0/4047696263 learned_addr learned my addr 172.21.15.114:0/4047696263 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:45.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35eaffd640 1 -- 172.21.15.114:0/4047696263 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35ec101a40 msgr2=0x7f35ec19c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:45.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35eaffd640 1 --2- 172.21.15.114:0/4047696263 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35ec101a40 0x7f35ec19c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:45.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35eaffd640 1 -- 172.21.15.114:0/4047696263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35ec19c7e0 con 0x7f35ec100840 2024-09-20T19:42:45.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35eaffd640 1 --2- 172.21.15.114:0/4047696263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35ec100840 0x7f35ec06a620 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f35dc00eaf0 tx=0x7f35dc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:45.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35f09e3640 1 -- 172.21.15.114:0/4047696263 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f35dc00cd10 con 0x7f35ec100840 2024-09-20T19:42:45.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35f09e3640 1 -- 172.21.15.114:0/4047696263 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f35dc003e30 con 0x7f35ec100840 2024-09-20T19:42:45.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35f19e5640 1 -- 172.21.15.114:0/4047696263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35ec19cad0 con 0x7f35ec100840 2024-09-20T19:42:45.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35f09e3640 1 -- 172.21.15.114:0/4047696263 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f35dc010660 con 0x7f35ec100840 2024-09-20T19:42:45.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.195+0000 7f35f19e5640 1 -- 172.21.15.114:0/4047696263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35ec19d030 con 0x7f35ec100840 2024-09-20T19:42:45.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.196+0000 7f35f19e5640 1 -- 172.21.15.114:0/4047696263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35ec100c40 con 0x7f35ec100840 2024-09-20T19:42:45.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.197+0000 7f35f09e3640 1 -- 172.21.15.114:0/4047696263 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f35dc003970 con 0x7f35ec100840 2024-09-20T19:42:45.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.197+0000 7f35f09e3640 1 --2- 172.21.15.114:0/4047696263 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f35d406e310 0x7f35d40707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:45.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.197+0000 7f35f09e3640 1 -- 172.21.15.114:0/4047696263 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f35dc08e920 con 0x7f35ec100840 2024-09-20T19:42:45.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.197+0000 7f35ea7fc640 1 --2- 172.21.15.114:0/4047696263 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f35d406e310 0x7f35d40707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:45.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.198+0000 7f35ea7fc640 1 --2- 172.21.15.114:0/4047696263 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f35d406e310 0x7f35d40707d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f35e0000c00 tx=0x7f35e00058d0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:45.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.202+0000 7f35f09e3640 1 -- 172.21.15.114:0/4047696263 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f35dc05c270 con 0x7f35ec100840 2024-09-20T19:42:45.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:44 smithi160 ceph-mon[35859]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-09-20T19:42:45.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:44 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/1017327713' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-09-20T19:42:45.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:44 smithi160 ceph-mon[35859]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 2 up:standby 2024-09-20T19:42:45.279 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:44 smithi160 ceph-mon[35859]: pgmap v178: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s wr, 4 op/s 2024-09-20T19:42:45.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.432+0000 7f35f19e5640 1 -- 172.21.15.114:0/4047696263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f35ec062df0 con 0x7f35ec100840 2024-09-20T19:42:45.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.432+0000 7f35f09e3640 1 -- 172.21.15.114:0/4047696263 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 10 v10) ==== 76+0+1803 (secure 0 0 0) 0x7f35dc05ff20 con 0x7f35ec100840 2024-09-20T19:42:45.434 INFO:teuthology.orchestra.run.smithi114.stdout:e10 2024-09-20T19:42:45.434 INFO:teuthology.orchestra.run.smithi114.stdout:btime 2024-09-20T19:42:43:879978+0000 2024-09-20T19:42:45.434 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T19:42:45.435 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-20T19:42:45.435 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-09-20T19:42:45.435 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:42:45.435 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-09-20T19:42:45.435 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-09-20T19:42:45.435 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 10 2024-09-20T19:42:45.435 INFO:teuthology.orchestra.run.smithi114.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-20T19:42:45.435 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-09-20T19:42:33.642256+0000 2024-09-20T19:42:45.435 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-09-20T19:42:43.291586+0000 2024-09-20T19:42:45.435 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-09-20T19:42:45.435 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-09-20T19:42:45.435 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-09-20T19:42:45.435 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-09-20T19:42:45.435 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-09-20T19:42:45.436 INFO:teuthology.orchestra.run.smithi114.stdout:max_xattr_size 65536 2024-09-20T19:42:45.436 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-09-20T19:42:45.436 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-09-20T19:42:45.436 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 0 2024-09-20T19:42:45.436 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-20T19:42:45.436 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 1 2024-09-20T19:42:45.436 INFO:teuthology.orchestra.run.smithi114.stdout:in 0 2024-09-20T19:42:45.436 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14550} 2024-09-20T19:42:45.436 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-09-20T19:42:45.436 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-09-20T19:42:45.436 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-09-20T19:42:45.436 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-09-20T19:42:45.436 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-09-20T19:42:45.436 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-09-20T19:42:45.436 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-09-20T19:42:45.437 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-09-20T19:42:45.437 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-09-20T19:42:45.437 INFO:teuthology.orchestra.run.smithi114.stdout:qdb_cluster leader: 14550 members: 14550 2024-09-20T19:42:45.437 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.scsmgq{0:14550} state up:active seq 2 addr [v2:172.21.15.114:6826/500124176,v1:172.21.15.114:6827/500124176] compat {c=[1],r=[1],i=[1fff]}] 2024-09-20T19:42:45.437 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:42:45.437 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:42:45.437 INFO:teuthology.orchestra.run.smithi114.stdout:Standby daemons: 2024-09-20T19:42:45.437 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:42:45.437 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tlrcnf{-1:14564} state up:standby seq 1 addr [v2:172.21.15.114:6828/3069148139,v1:172.21.15.114:6829/3069148139] compat {c=[1],r=[1],i=[1fff]}] 2024-09-20T19:42:45.437 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi160.xirihp{-1:24287} state up:standby seq 1 addr [v2:172.21.15.160:6824/2564850174,v1:172.21.15.160:6825/2564850174] compat {c=[1],r=[1],i=[1fff]}] 2024-09-20T19:42:45.438 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 10 2024-09-20T19:42:45.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.436+0000 7f35f19e5640 1 -- 172.21.15.114:0/4047696263 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f35d406e310 msgr2=0x7f35d40707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:45.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.436+0000 7f35f19e5640 1 --2- 172.21.15.114:0/4047696263 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f35d406e310 0x7f35d40707d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f35e0000c00 tx=0x7f35e00058d0 comp rx=0 tx=0).stop 2024-09-20T19:42:45.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.436+0000 7f35f19e5640 1 -- 172.21.15.114:0/4047696263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35ec100840 msgr2=0x7f35ec06a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:45.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.436+0000 7f35f19e5640 1 --2- 172.21.15.114:0/4047696263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35ec100840 0x7f35ec06a620 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f35dc00eaf0 tx=0x7f35dc00c2d0 comp rx=0 tx=0).stop 2024-09-20T19:42:45.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.436+0000 7f35f19e5640 1 -- 172.21.15.114:0/4047696263 shutdown_connections 2024-09-20T19:42:45.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.436+0000 7f35f19e5640 1 --2- 172.21.15.114:0/4047696263 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f35d406e310 0x7f35d40707d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:45.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.436+0000 7f35f19e5640 1 --2- 172.21.15.114:0/4047696263 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f35ec101a40 0x7f35ec19c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:45.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.436+0000 7f35f19e5640 1 --2- 172.21.15.114:0/4047696263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35ec100840 0x7f35ec06a620 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:45.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.436+0000 7f35f19e5640 1 -- 172.21.15.114:0/4047696263 >> 172.21.15.114:0/4047696263 conn(0x7f35ec0fbff0 msgr2=0x7f35ec104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:45.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.436+0000 7f35f19e5640 1 -- 172.21.15.114:0/4047696263 shutdown_connections 2024-09-20T19:42:45.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:45.436+0000 7f35f19e5640 1 -- 172.21.15.114:0/4047696263 wait complete. 2024-09-20T19:42:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:46 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:46 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:46 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:46 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:46.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:46 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:46.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:46 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:42:46.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:46 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/4047696263' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-20T19:42:46.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:46 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:46.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:46 smithi114 ceph-mon[28260]: mds.? [v2:172.21.15.160:6826/1537707856,v1:172.21.15.160:6827/1537707856] up:boot 2024-09-20T19:42:46.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:46 smithi114 ceph-mon[28260]: mds.? [v2:172.21.15.114:6826/500124176,v1:172.21.15.114:6827/500124176] up:active 2024-09-20T19:42:46.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:46 smithi114 ceph-mon[28260]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 3 up:standby 2024-09-20T19:42:46.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:46 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi160.lylyja"}]: dispatch 2024-09-20T19:42:46.353 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-09-20T19:42:46.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:46 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:46.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:46 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:46.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:46 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:46.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:46 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:46.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:46 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:46.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:46 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:42:46.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:46 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/4047696263' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-20T19:42:46.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:46 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:46.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:46 smithi160 ceph-mon[35859]: mds.? [v2:172.21.15.160:6826/1537707856,v1:172.21.15.160:6827/1537707856] up:boot 2024-09-20T19:42:46.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:46 smithi160 ceph-mon[35859]: mds.? [v2:172.21.15.114:6826/500124176,v1:172.21.15.114:6827/500124176] up:active 2024-09-20T19:42:46.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:46 smithi160 ceph-mon[35859]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 3 up:standby 2024-09-20T19:42:46.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:46 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi160.lylyja"}]: dispatch 2024-09-20T19:42:47.022 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:47.432 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:47 smithi114 ceph-mon[28260]: pgmap v179: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-09-20T19:42:47.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:47 smithi160 ceph-mon[35859]: pgmap v179: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-09-20T19:42:47.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.511+0000 7f88a72b5640 1 -- 172.21.15.114:0/1960948909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88a0100880 msgr2=0x7f88a0100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:47.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.511+0000 7f88a72b5640 1 --2- 172.21.15.114:0/1960948909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88a0100880 0x7f88a0100c80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8890009c20 tx=0x7f889002a970 comp rx=0 tx=0).stop 2024-09-20T19:42:47.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.512+0000 7f88a72b5640 1 -- 172.21.15.114:0/1960948909 shutdown_connections 2024-09-20T19:42:47.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.512+0000 7f88a72b5640 1 --2- 172.21.15.114:0/1960948909 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88a0101a80 0x7f88a0101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:47.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.512+0000 7f88a72b5640 1 --2- 172.21.15.114:0/1960948909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88a0100880 0x7f88a0100c80 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:47.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.512+0000 7f88a72b5640 1 -- 172.21.15.114:0/1960948909 >> 172.21.15.114:0/1960948909 conn(0x7f88a00fc010 msgr2=0x7f88a00fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:47.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.512+0000 7f88a72b5640 1 -- 172.21.15.114:0/1960948909 shutdown_connections 2024-09-20T19:42:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.512+0000 7f88a72b5640 1 -- 172.21.15.114:0/1960948909 wait complete. 2024-09-20T19:42:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.512+0000 7f88a72b5640 1 Processor -- start 2024-09-20T19:42:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.512+0000 7f88a72b5640 1 -- start start 2024-09-20T19:42:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.512+0000 7f88a72b5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88a0100880 0x7f88a006adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.512+0000 7f88a72b5640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88a0101a80 0x7f88a019e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.512+0000 7f88a72b5640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f88a01096a0 con 0x7f88a0100880 2024-09-20T19:42:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.512+0000 7f88a72b5640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f88a0109520 con 0x7f88a0101a80 2024-09-20T19:42:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.512+0000 7f88a5051640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88a0100880 0x7f88a006adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.512+0000 7f88a5051640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88a0100880 0x7f88a006adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60900/0 (socket says 172.21.15.114:60900) 2024-09-20T19:42:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.512+0000 7f88a5051640 1 -- 172.21.15.114:0/3611494655 learned_addr learned my addr 172.21.15.114:0/3611494655 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.513+0000 7f88a4850640 1 --2- 172.21.15.114:0/3611494655 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88a0101a80 0x7f88a019e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.513+0000 7f88a5051640 1 -- 172.21.15.114:0/3611494655 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88a0101a80 msgr2=0x7f88a019e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.513+0000 7f88a5051640 1 --2- 172.21.15.114:0/3611494655 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88a0101a80 0x7f88a019e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:47.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.513+0000 7f88a5051640 1 -- 172.21.15.114:0/3611494655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a019eb30 con 0x7f88a0100880 2024-09-20T19:42:47.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.513+0000 7f88a5051640 1 --2- 172.21.15.114:0/3611494655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88a0100880 0x7f88a006adf0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8890006010 tx=0x7f889002f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:47.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.513+0000 7f889e7fc640 1 -- 172.21.15.114:0/3611494655 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8890047070 con 0x7f88a0100880 2024-09-20T19:42:47.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.513+0000 7f88a72b5640 1 -- 172.21.15.114:0/3611494655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a019ed00 con 0x7f88a0100880 2024-09-20T19:42:47.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.513+0000 7f889e7fc640 1 -- 172.21.15.114:0/3611494655 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8890003e30 con 0x7f88a0100880 2024-09-20T19:42:47.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.513+0000 7f889e7fc640 1 -- 172.21.15.114:0/3611494655 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8890034070 con 0x7f88a0100880 2024-09-20T19:42:47.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.513+0000 7f88a72b5640 1 -- 172.21.15.114:0/3611494655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a019f1b0 con 0x7f88a0100880 2024-09-20T19:42:47.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.514+0000 7f88a72b5640 1 -- 172.21.15.114:0/3611494655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a0100c80 con 0x7f88a0100880 2024-09-20T19:42:47.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.514+0000 7f889e7fc640 1 -- 172.21.15.114:0/3611494655 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f889004a070 con 0x7f88a0100880 2024-09-20T19:42:47.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.515+0000 7f889e7fc640 1 --2- 172.21.15.114:0/3611494655 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f888806e2b0 0x7f8888070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:47.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.515+0000 7f889e7fc640 1 -- 172.21.15.114:0/3611494655 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f88900aee90 con 0x7f88a0100880 2024-09-20T19:42:47.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.515+0000 7f88a4850640 1 --2- 172.21.15.114:0/3611494655 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f888806e2b0 0x7f8888070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:47.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.515+0000 7f88a4850640 1 --2- 172.21.15.114:0/3611494655 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f888806e2b0 0x7f8888070770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f88a006ab70 tx=0x7f8894005f90 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:47.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.519+0000 7f889e7fc640 1 -- 172.21.15.114:0/3611494655 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f889003f070 con 0x7f88a0100880 2024-09-20T19:42:47.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.732+0000 7f88a72b5640 1 -- 172.21.15.114:0/3611494655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88a019fb20 con 0x7f88a0100880 2024-09-20T19:42:47.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.733+0000 7f889e7fc640 1 -- 172.21.15.114:0/3611494655 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+5324 (secure 0 0 0) 0x7f889007c880 con 0x7f88a0100880 2024-09-20T19:42:47.735 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 12 2024-09-20T19:42:47.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.735+0000 7f88a72b5640 1 -- 172.21.15.114:0/3611494655 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f888806e2b0 msgr2=0x7f8888070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:47.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.735+0000 7f88a72b5640 1 --2- 172.21.15.114:0/3611494655 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f888806e2b0 0x7f8888070770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f88a006ab70 tx=0x7f8894005f90 comp rx=0 tx=0).stop 2024-09-20T19:42:47.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.735+0000 7f88a72b5640 1 -- 172.21.15.114:0/3611494655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88a0100880 msgr2=0x7f88a006adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:47.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.735+0000 7f88a72b5640 1 --2- 172.21.15.114:0/3611494655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88a0100880 0x7f88a006adf0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8890006010 tx=0x7f889002f5d0 comp rx=0 tx=0).stop 2024-09-20T19:42:47.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.735+0000 7f88a72b5640 1 -- 172.21.15.114:0/3611494655 shutdown_connections 2024-09-20T19:42:47.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.735+0000 7f88a72b5640 1 --2- 172.21.15.114:0/3611494655 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f888806e2b0 0x7f8888070770 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:47.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.735+0000 7f88a72b5640 1 --2- 172.21.15.114:0/3611494655 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88a0101a80 0x7f88a019e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:47.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.735+0000 7f88a72b5640 1 --2- 172.21.15.114:0/3611494655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88a0100880 0x7f88a006adf0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:47.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.735+0000 7f88a72b5640 1 -- 172.21.15.114:0/3611494655 >> 172.21.15.114:0/3611494655 conn(0x7f88a00fc010 msgr2=0x7f88a00fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:47.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.735+0000 7f88a72b5640 1 -- 172.21.15.114:0/3611494655 shutdown_connections 2024-09-20T19:42:47.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:47.735+0000 7f88a72b5640 1 -- 172.21.15.114:0/3611494655 wait complete. 2024-09-20T19:42:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-20T19:42:48.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:48 smithi114 ceph-mon[28260]: mds.? [v2:172.21.15.114:6828/3069148139,v1:172.21.15.114:6829/3069148139] up:standby 2024-09-20T19:42:48.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:48 smithi114 ceph-mon[28260]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 3 up:standby 2024-09-20T19:42:48.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:48 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:48.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:48 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:48.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:48 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3611494655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-20T19:42:48.506 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-09-20T19:42:48.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:48 smithi160 ceph-mon[35859]: mds.? [v2:172.21.15.114:6828/3069148139,v1:172.21.15.114:6829/3069148139] up:standby 2024-09-20T19:42:48.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:48 smithi160 ceph-mon[35859]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 3 up:standby 2024-09-20T19:42:48.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:48 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:48.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:48 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:48.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:48 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/3611494655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-20T19:42:49.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:49 smithi114 ceph-mon[28260]: pgmap v180: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-20T19:42:49.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:49 smithi114 ceph-mon[28260]: mds.? [v2:172.21.15.160:6824/2564850174,v1:172.21.15.160:6825/2564850174] up:standby 2024-09-20T19:42:49.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:49 smithi114 ceph-mon[28260]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 3 up:standby 2024-09-20T19:42:49.748 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:49.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:49 smithi160 ceph-mon[35859]: pgmap v180: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-20T19:42:49.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:49 smithi160 ceph-mon[35859]: mds.? [v2:172.21.15.160:6824/2564850174,v1:172.21.15.160:6825/2564850174] up:standby 2024-09-20T19:42:49.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:49 smithi160 ceph-mon[35859]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 3 up:standby 2024-09-20T19:42:50.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:50 smithi114 ceph-mon[28260]: mds.? [v2:172.21.15.160:6826/1537707856,v1:172.21.15.160:6827/1537707856] up:standby 2024-09-20T19:42:50.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:50 smithi114 ceph-mon[28260]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 3 up:standby 2024-09-20T19:42:50.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:50 smithi160 ceph-mon[35859]: mds.? [v2:172.21.15.160:6826/1537707856,v1:172.21.15.160:6827/1537707856] up:standby 2024-09-20T19:42:50.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:50 smithi160 ceph-mon[35859]: fsmap cephfs:1 {0=cephfs.smithi114.scsmgq=up:active} 3 up:standby 2024-09-20T19:42:50.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.866+0000 7f0d2ba8e640 1 -- 172.21.15.114:0/1085337542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d24101a40 msgr2=0x7f0d24101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:50.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.866+0000 7f0d2ba8e640 1 --2- 172.21.15.114:0/1085337542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d24101a40 0x7f0d24101ec0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0d18009c20 tx=0x7f0d1802a970 comp rx=0 tx=0).stop 2024-09-20T19:42:50.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.867+0000 7f0d2ba8e640 1 -- 172.21.15.114:0/1085337542 shutdown_connections 2024-09-20T19:42:50.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.867+0000 7f0d2ba8e640 1 --2- 172.21.15.114:0/1085337542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d24101a40 0x7f0d24101ec0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:50.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.867+0000 7f0d2ba8e640 1 --2- 172.21.15.114:0/1085337542 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0d24100840 0x7f0d24100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:50.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.867+0000 7f0d2ba8e640 1 -- 172.21.15.114:0/1085337542 >> 172.21.15.114:0/1085337542 conn(0x7f0d240fbff0 msgr2=0x7f0d240fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:50.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.867+0000 7f0d2ba8e640 1 -- 172.21.15.114:0/1085337542 shutdown_connections 2024-09-20T19:42:50.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.867+0000 7f0d2ba8e640 1 -- 172.21.15.114:0/1085337542 wait complete. 2024-09-20T19:42:50.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.867+0000 7f0d2ba8e640 1 Processor -- start 2024-09-20T19:42:50.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.867+0000 7f0d2ba8e640 1 -- start start 2024-09-20T19:42:50.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.867+0000 7f0d2ba8e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0d24100840 0x7f0d2406a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:50.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.867+0000 7f0d2ba8e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d24101a40 0x7f0d2419c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:50.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.867+0000 7f0d2ba8e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0d24109700 con 0x7f0d24101a40 2024-09-20T19:42:50.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.867+0000 7f0d2ba8e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0d24109580 con 0x7f0d24100840 2024-09-20T19:42:50.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.867+0000 7f0d29029640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d24101a40 0x7f0d2419c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:50.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.867+0000 7f0d29029640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d24101a40 0x7f0d2419c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60928/0 (socket says 172.21.15.114:60928) 2024-09-20T19:42:50.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.867+0000 7f0d29029640 1 -- 172.21.15.114:0/2915574317 learned_addr learned my addr 172.21.15.114:0/2915574317 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:50.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.868+0000 7f0d2982a640 1 --2- 172.21.15.114:0/2915574317 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0d24100840 0x7f0d2406a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:50.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.868+0000 7f0d29029640 1 -- 172.21.15.114:0/2915574317 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0d24100840 msgr2=0x7f0d2406a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:50.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.868+0000 7f0d29029640 1 --2- 172.21.15.114:0/2915574317 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0d24100840 0x7f0d2406a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:50.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.868+0000 7f0d29029640 1 -- 172.21.15.114:0/2915574317 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d2419c8f0 con 0x7f0d24101a40 2024-09-20T19:42:50.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.868+0000 7f0d2982a640 1 --2- 172.21.15.114:0/2915574317 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0d24100840 0x7f0d2406a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-20T19:42:50.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.868+0000 7f0d29029640 1 --2- 172.21.15.114:0/2915574317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d24101a40 0x7f0d2419c300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0d1802f900 tx=0x7f0d1802f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:50.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.868+0000 7f0d22ffd640 1 -- 172.21.15.114:0/2915574317 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0d18047070 con 0x7f0d24101a40 2024-09-20T19:42:50.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.868+0000 7f0d22ffd640 1 -- 172.21.15.114:0/2915574317 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0d18003e30 con 0x7f0d24101a40 2024-09-20T19:42:50.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.868+0000 7f0d2ba8e640 1 -- 172.21.15.114:0/2915574317 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d2419cb80 con 0x7f0d24101a40 2024-09-20T19:42:50.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.868+0000 7f0d22ffd640 1 -- 172.21.15.114:0/2915574317 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0d18034070 con 0x7f0d24101a40 2024-09-20T19:42:50.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.868+0000 7f0d2ba8e640 1 -- 172.21.15.114:0/2915574317 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d2419d130 con 0x7f0d24101a40 2024-09-20T19:42:50.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.869+0000 7f0d2ba8e640 1 -- 172.21.15.114:0/2915574317 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d24100cc0 con 0x7f0d24101a40 2024-09-20T19:42:50.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.870+0000 7f0d22ffd640 1 -- 172.21.15.114:0/2915574317 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f0d1804a070 con 0x7f0d24101a40 2024-09-20T19:42:50.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.870+0000 7f0d22ffd640 1 --2- 172.21.15.114:0/2915574317 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0d0c06e2b0 0x7f0d0c070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:50.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.870+0000 7f0d22ffd640 1 -- 172.21.15.114:0/2915574317 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f0d180af0f0 con 0x7f0d24101a40 2024-09-20T19:42:50.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.870+0000 7f0d2982a640 1 --2- 172.21.15.114:0/2915574317 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0d0c06e2b0 0x7f0d0c070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:50.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.871+0000 7f0d2982a640 1 --2- 172.21.15.114:0/2915574317 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0d0c06e2b0 0x7f0d0c070770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0d241018a0 tx=0x7f0d1c008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:50.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:50.875+0000 7f0d22ffd640 1 -- 172.21.15.114:0/2915574317 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f0d1803f070 con 0x7f0d24101a40 2024-09-20T19:42:51.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:51.139+0000 7f0d2ba8e640 1 -- 172.21.15.114:0/2915574317 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) -- 0x7f0d24062df0 con 0x7f0d24101a40 2024-09-20T19:42:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:51.139+0000 7f0d22ffd640 1 -- 172.21.15.114:0/2915574317 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v14) ==== 78+0+98 (secure 0 0 0) 0x7f0d1802fe40 con 0x7f0d24101a40 2024-09-20T19:42:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:51.141+0000 7f0d2ba8e640 1 -- 172.21.15.114:0/2915574317 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0d0c06e2b0 msgr2=0x7f0d0c070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:51.141+0000 7f0d2ba8e640 1 --2- 172.21.15.114:0/2915574317 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0d0c06e2b0 0x7f0d0c070770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0d241018a0 tx=0x7f0d1c008040 comp rx=0 tx=0).stop 2024-09-20T19:42:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:51.141+0000 7f0d2ba8e640 1 -- 172.21.15.114:0/2915574317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d24101a40 msgr2=0x7f0d2419c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:51.141+0000 7f0d2ba8e640 1 --2- 172.21.15.114:0/2915574317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d24101a40 0x7f0d2419c300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0d1802f900 tx=0x7f0d1802f730 comp rx=0 tx=0).stop 2024-09-20T19:42:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:51.141+0000 7f0d2ba8e640 1 -- 172.21.15.114:0/2915574317 shutdown_connections 2024-09-20T19:42:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:51.141+0000 7f0d2ba8e640 1 --2- 172.21.15.114:0/2915574317 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0d0c06e2b0 0x7f0d0c070770 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:51.141+0000 7f0d2ba8e640 1 --2- 172.21.15.114:0/2915574317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d24101a40 0x7f0d2419c300 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:51.141+0000 7f0d2ba8e640 1 --2- 172.21.15.114:0/2915574317 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0d24100840 0x7f0d2406a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:51.142+0000 7f0d2ba8e640 1 -- 172.21.15.114:0/2915574317 >> 172.21.15.114:0/2915574317 conn(0x7f0d240fbff0 msgr2=0x7f0d24104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:51.142+0000 7f0d2ba8e640 1 -- 172.21.15.114:0/2915574317 shutdown_connections 2024-09-20T19:42:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:51.142+0000 7f0d2ba8e640 1 -- 172.21.15.114:0/2915574317 wait complete. 2024-09-20T19:42:51.152 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-20T19:42:51.658 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:51 smithi114 ceph-mon[28260]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-20T19:42:51.658 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:51 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/2915574317' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-09-20T19:42:51.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:51 smithi160 ceph-mon[35859]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-20T19:42:51.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:51 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/2915574317' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-09-20T19:42:52.010 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-09-20T19:42:52.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-20T19:42:52.297 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:52.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.767+0000 7f5dd11ac640 1 -- 172.21.15.114:0/2738744192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dcc107910 msgr2=0x7f5dcc107cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:52.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.767+0000 7f5dd11ac640 1 --2- 172.21.15.114:0/2738744192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dcc107910 0x7f5dcc107cf0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc009880 tx=0x7f5dbc02a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:52.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.767+0000 7f5dd11ac640 1 -- 172.21.15.114:0/2738744192 shutdown_connections 2024-09-20T19:42:52.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.767+0000 7f5dd11ac640 1 --2- 172.21.15.114:0/2738744192 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5dcc101910 0x7f5dcc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:52.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.767+0000 7f5dd11ac640 1 --2- 172.21.15.114:0/2738744192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dcc107910 0x7f5dcc107cf0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:52.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.767+0000 7f5dd11ac640 1 -- 172.21.15.114:0/2738744192 >> 172.21.15.114:0/2738744192 conn(0x7f5dcc0fd5f0 msgr2=0x7f5dcc0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:52.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.767+0000 7f5dd11ac640 1 -- 172.21.15.114:0/2738744192 shutdown_connections 2024-09-20T19:42:52.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.767+0000 7f5dd11ac640 1 -- 172.21.15.114:0/2738744192 wait complete. 2024-09-20T19:42:52.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.768+0000 7f5dd11ac640 1 Processor -- start 2024-09-20T19:42:52.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.768+0000 7f5dd11ac640 1 -- start start 2024-09-20T19:42:52.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.768+0000 7f5dd11ac640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dcc101910 0x7f5dcc19d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:52.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.768+0000 7f5dd11ac640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5dcc107910 0x7f5dcc19dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:52.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.768+0000 7f5dd11ac640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc10ede0 con 0x7f5dcc101910 2024-09-20T19:42:52.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.768+0000 7f5dd11ac640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc10ec60 con 0x7f5dcc107910 2024-09-20T19:42:52.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.768+0000 7f5dcad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dcc101910 0x7f5dcc19d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:52.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.768+0000 7f5dcad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dcc101910 0x7f5dcc19d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60940/0 (socket says 172.21.15.114:60940) 2024-09-20T19:42:52.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.768+0000 7f5dcad9d640 1 -- 172.21.15.114:0/904290132 learned_addr learned my addr 172.21.15.114:0/904290132 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:52.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.768+0000 7f5dca59c640 1 --2- 172.21.15.114:0/904290132 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5dcc107910 0x7f5dcc19dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:52.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.769+0000 7f5dcad9d640 1 -- 172.21.15.114:0/904290132 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5dcc107910 msgr2=0x7f5dcc19dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:52.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.769+0000 7f5dcad9d640 1 --2- 172.21.15.114:0/904290132 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5dcc107910 0x7f5dcc19dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:52.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.769+0000 7f5dcad9d640 1 -- 172.21.15.114:0/904290132 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dcc197810 con 0x7f5dcc101910 2024-09-20T19:42:52.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.769+0000 7f5dcad9d640 1 --2- 172.21.15.114:0/904290132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dcc101910 0x7f5dcc19d690 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc031460 tx=0x7f5dbc0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:52.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.769+0000 7f5db3fff640 1 -- 172.21.15.114:0/904290132 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5dbc03f070 con 0x7f5dcc101910 2024-09-20T19:42:52.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.769+0000 7f5dd11ac640 1 -- 172.21.15.114:0/904290132 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dcc197aa0 con 0x7f5dcc101910 2024-09-20T19:42:52.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.769+0000 7f5db3fff640 1 -- 172.21.15.114:0/904290132 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5dbc004cf0 con 0x7f5dcc101910 2024-09-20T19:42:52.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.769+0000 7f5db3fff640 1 -- 172.21.15.114:0/904290132 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5dbc035030 con 0x7f5dcc101910 2024-09-20T19:42:52.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.769+0000 7f5dd11ac640 1 -- 172.21.15.114:0/904290132 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dcc197fd0 con 0x7f5dcc101910 2024-09-20T19:42:52.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.770+0000 7f5dd11ac640 1 -- 172.21.15.114:0/904290132 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dcc103050 con 0x7f5dcc101910 2024-09-20T19:42:52.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.770+0000 7f5db3fff640 1 -- 172.21.15.114:0/904290132 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f5dbc042070 con 0x7f5dcc101910 2024-09-20T19:42:52.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.771+0000 7f5db3fff640 1 --2- 172.21.15.114:0/904290132 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5db406e2b0 0x7f5db4070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:52.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.771+0000 7f5dca59c640 1 --2- 172.21.15.114:0/904290132 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5db406e2b0 0x7f5db4070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:52.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.771+0000 7f5db3fff640 1 -- 172.21.15.114:0/904290132 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f5dbc0aeb30 con 0x7f5dcc101910 2024-09-20T19:42:52.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.772+0000 7f5dca59c640 1 --2- 172.21.15.114:0/904290132 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5db406e2b0 0x7f5db4070770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc0ffc10 tx=0x7f5dc000a250 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:52.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.775+0000 7f5db3fff640 1 -- 172.21.15.114:0/904290132 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f5dbc080140 con 0x7f5dcc101910 2024-09-20T19:42:52.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.994+0000 7f5dd11ac640 1 -- 172.21.15.114:0/904290132 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5dcc109e80 con 0x7f5dcc101910 2024-09-20T19:42:52.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.995+0000 7f5db3fff640 1 -- 172.21.15.114:0/904290132 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+5323 (secure 0 0 0) 0x7f5dbc084970 con 0x7f5dcc101910 2024-09-20T19:42:52.997 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-20T19:42:52.997 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":14,"btime":"2024-09-20T19:42:49:297354+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14564,"name":"cephfs.smithi114.tlrcnf","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.114:6829/3069148139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3069148139},{"type":"v1","addr":"172.21.15.114:6829","nonce":3069148139}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24287,"name":"cephfs.smithi160.xirihp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.160:6825/2564850174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":2564850174},{"type":"v1","addr":"172.21.15.160:6825","nonce":2564850174}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24299,"name":"cephfs.smithi160.lylyja","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.160:6827/1537707856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":1537707856},{"type":"v1","addr":"172.21.15.160:6827","nonce":1537707856}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-20T19:42:33.642256+0000","modified":"2024-09-20T19:42:45.677202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":14550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14550":{"gid":14550,"name":"cephfs.smithi114.scsmgq","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6827/500124176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":500124176},{"type":"v1","addr":"172.21.15.114:6827","nonce":500124176}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14550,"qdb_cluster":[14550]},"id":1}]} 2024-09-20T19:42:52.997 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 14 2024-09-20T19:42:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.997+0000 7f5dd11ac640 1 -- 172.21.15.114:0/904290132 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5db406e2b0 msgr2=0x7f5db4070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.997+0000 7f5dd11ac640 1 --2- 172.21.15.114:0/904290132 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5db406e2b0 0x7f5db4070770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc0ffc10 tx=0x7f5dc000a250 comp rx=0 tx=0).stop 2024-09-20T19:42:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.997+0000 7f5dd11ac640 1 -- 172.21.15.114:0/904290132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dcc101910 msgr2=0x7f5dcc19d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.997+0000 7f5dd11ac640 1 --2- 172.21.15.114:0/904290132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dcc101910 0x7f5dcc19d690 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc031460 tx=0x7f5dbc0048f0 comp rx=0 tx=0).stop 2024-09-20T19:42:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.997+0000 7f5dd11ac640 1 -- 172.21.15.114:0/904290132 shutdown_connections 2024-09-20T19:42:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.997+0000 7f5dd11ac640 1 --2- 172.21.15.114:0/904290132 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5db406e2b0 0x7f5db4070770 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.997+0000 7f5dd11ac640 1 --2- 172.21.15.114:0/904290132 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5dcc107910 0x7f5dcc19dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.997+0000 7f5dd11ac640 1 --2- 172.21.15.114:0/904290132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dcc101910 0x7f5dcc19d690 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.997+0000 7f5dd11ac640 1 -- 172.21.15.114:0/904290132 >> 172.21.15.114:0/904290132 conn(0x7f5dcc0fd5f0 msgr2=0x7f5dcc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.997+0000 7f5dd11ac640 1 -- 172.21.15.114:0/904290132 shutdown_connections 2024-09-20T19:42:52.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:52.997+0000 7f5dd11ac640 1 -- 172.21.15.114:0/904290132 wait complete. 2024-09-20T19:42:53.471 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:53 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:53.471 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:53 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:53.473 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:53 smithi160 ceph-mon[35859]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 2 op/s 2024-09-20T19:42:53.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:53 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:42:53.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:53 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:42:53.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:53 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:53.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:53 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:42:53.474 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:53 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/904290132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-20T19:42:53.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:53 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:53.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:53 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:53.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:53 smithi114 ceph-mon[28260]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 2 op/s 2024-09-20T19:42:53.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:53 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:42:53.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:53 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:42:53.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:53 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:53.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:53 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:42:53.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:53 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/904290132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-20T19:42:53.602 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 11, 'max_mds': 1, 'flags': 18} 2024-09-20T19:42:53.602 INFO:teuthology.run_tasks:Running task kclient... 2024-09-20T19:42:53.626 INFO:tasks.kclient:Mounting kernel clients... 2024-09-20T19:42:53.626 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-09-20T19:42:53.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:42:53.626 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-09-20T19:42:53.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:42:53.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-09-20T19:42:53.712 INFO:teuthology.orchestra.run.smithi114.stderr:Cannot find device "ceph-brx" 2024-09-20T19:42:53.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:42:53.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:42:53.714 DEBUG:teuthology.orchestra.run.smithi160:> ip netns list 2024-09-20T19:42:53.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:42:53.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo ip link delete ceph-brx 2024-09-20T19:42:53.798 INFO:teuthology.orchestra.run.smithi160.stderr:Cannot find device "ceph-brx" 2024-09-20T19:42:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T19:42:53.800 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-09-20T19:42:53.800 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-20T19:42:53.800 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-20T19:42:53.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph fs ls 2024-09-20T19:42:54.470 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.907+0000 7ff0c77fe640 1 -- 172.21.15.114:0/898837057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0c01033c0 msgr2=0x7ff0c010b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.907+0000 7ff0c77fe640 1 --2- 172.21.15.114:0/898837057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0c01033c0 0x7ff0c010b8b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4009880 tx=0x7ff0b402a530 comp rx=0 tx=0).stop 2024-09-20T19:42:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.907+0000 7ff0c77fe640 1 -- 172.21.15.114:0/898837057 shutdown_connections 2024-09-20T19:42:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.907+0000 7ff0c77fe640 1 --2- 172.21.15.114:0/898837057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0c01033c0 0x7ff0c010b8b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.907+0000 7ff0c77fe640 1 --2- 172.21.15.114:0/898837057 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0c0102aa0 0x7ff0c0102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.907+0000 7ff0c77fe640 1 -- 172.21.15.114:0/898837057 >> 172.21.15.114:0/898837057 conn(0x7ff0c00fc4c0 msgr2=0x7ff0c00fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:54.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.908+0000 7ff0c77fe640 1 -- 172.21.15.114:0/898837057 shutdown_connections 2024-09-20T19:42:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.908+0000 7ff0c77fe640 1 -- 172.21.15.114:0/898837057 wait complete. 2024-09-20T19:42:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.908+0000 7ff0c77fe640 1 Processor -- start 2024-09-20T19:42:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.908+0000 7ff0c77fe640 1 -- start start 2024-09-20T19:42:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.909+0000 7ff0c77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0c0102aa0 0x7ff0c019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.909+0000 7ff0c77fe640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0c01033c0 0x7ff0c019db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.909+0000 7ff0c77fe640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c010edb0 con 0x7ff0c0102aa0 2024-09-20T19:42:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.909+0000 7ff0c77fe640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c010ec30 con 0x7ff0c01033c0 2024-09-20T19:42:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.909+0000 7ff0c559a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0c0102aa0 0x7ff0c019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.909+0000 7ff0c559a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0c0102aa0 0x7ff0c019d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60952/0 (socket says 172.21.15.114:60952) 2024-09-20T19:42:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.909+0000 7ff0c559a640 1 -- 172.21.15.114:0/3267801201 learned_addr learned my addr 172.21.15.114:0/3267801201 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.909+0000 7ff0c4d99640 1 --2- 172.21.15.114:0/3267801201 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0c01033c0 0x7ff0c019db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.909+0000 7ff0c559a640 1 -- 172.21.15.114:0/3267801201 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0c01033c0 msgr2=0x7ff0c019db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.909+0000 7ff0c559a640 1 --2- 172.21.15.114:0/3267801201 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0c01033c0 0x7ff0c019db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:54.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.909+0000 7ff0c559a640 1 -- 172.21.15.114:0/3267801201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c01977c0 con 0x7ff0c0102aa0 2024-09-20T19:42:54.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.909+0000 7ff0c559a640 1 --2- 172.21.15.114:0/3267801201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0c0102aa0 0x7ff0c019d640 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00e970 tx=0x7ff0bc00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:54.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.909+0000 7ff0b27fc640 1 -- 172.21.15.114:0/3267801201 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff0bc00cac0 con 0x7ff0c0102aa0 2024-09-20T19:42:54.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.909+0000 7ff0b27fc640 1 -- 172.21.15.114:0/3267801201 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff0bc00cc60 con 0x7ff0c0102aa0 2024-09-20T19:42:54.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.909+0000 7ff0c77fe640 1 -- 172.21.15.114:0/3267801201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c0197ab0 con 0x7ff0c0102aa0 2024-09-20T19:42:54.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.910+0000 7ff0b27fc640 1 -- 172.21.15.114:0/3267801201 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff0bc018720 con 0x7ff0c0102aa0 2024-09-20T19:42:54.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.910+0000 7ff0c77fe640 1 -- 172.21.15.114:0/3267801201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c0198010 con 0x7ff0c0102aa0 2024-09-20T19:42:54.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.910+0000 7ff0c77fe640 1 -- 172.21.15.114:0/3267801201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0c006a410 con 0x7ff0c0102aa0 2024-09-20T19:42:54.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.911+0000 7ff0b27fc640 1 -- 172.21.15.114:0/3267801201 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7ff0bc010430 con 0x7ff0c0102aa0 2024-09-20T19:42:54.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.911+0000 7ff0b27fc640 1 --2- 172.21.15.114:0/3267801201 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff0ac06e310 0x7ff0ac0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:54.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.912+0000 7ff0c4d99640 1 --2- 172.21.15.114:0/3267801201 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff0ac06e310 0x7ff0ac0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:54.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.912+0000 7ff0b27fc640 1 -- 172.21.15.114:0/3267801201 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7ff0bc08e630 con 0x7ff0c0102aa0 2024-09-20T19:42:54.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.912+0000 7ff0c4d99640 1 --2- 172.21.15.114:0/3267801201 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff0ac06e310 0x7ff0ac0707d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0198b10 tx=0x7ff0b4031400 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:54.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:54.916+0000 7ff0b27fc640 1 -- 172.21.15.114:0/3267801201 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7ff0bc05fcc0 con 0x7ff0c0102aa0 2024-09-20T19:42:55.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:55.136+0000 7ff0c77fe640 1 -- 172.21.15.114:0/3267801201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff0c0105fb0 con 0x7ff0c0102aa0 2024-09-20T19:42:55.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:55.136+0000 7ff0b27fc640 1 -- 172.21.15.114:0/3267801201 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) ==== 53+0+83 (secure 0 0 0) 0x7ff0bc0644f0 con 0x7ff0c0102aa0 2024-09-20T19:42:55.138 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-09-20T19:42:55.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:55.138+0000 7ff0c77fe640 1 -- 172.21.15.114:0/3267801201 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff0ac06e310 msgr2=0x7ff0ac0707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:55.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:55.138+0000 7ff0c77fe640 1 --2- 172.21.15.114:0/3267801201 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff0ac06e310 0x7ff0ac0707d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0198b10 tx=0x7ff0b4031400 comp rx=0 tx=0).stop 2024-09-20T19:42:55.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:55.138+0000 7ff0c77fe640 1 -- 172.21.15.114:0/3267801201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0c0102aa0 msgr2=0x7ff0c019d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:55.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:55.138+0000 7ff0c77fe640 1 --2- 172.21.15.114:0/3267801201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0c0102aa0 0x7ff0c019d640 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00e970 tx=0x7ff0bc00ee30 comp rx=0 tx=0).stop 2024-09-20T19:42:55.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:55.138+0000 7ff0c77fe640 1 -- 172.21.15.114:0/3267801201 shutdown_connections 2024-09-20T19:42:55.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:55.138+0000 7ff0c77fe640 1 --2- 172.21.15.114:0/3267801201 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7ff0ac06e310 0x7ff0ac0707d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:55.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:55.138+0000 7ff0c77fe640 1 --2- 172.21.15.114:0/3267801201 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff0c01033c0 0x7ff0c019db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:55.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:55.138+0000 7ff0c77fe640 1 --2- 172.21.15.114:0/3267801201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0c0102aa0 0x7ff0c019d640 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:55.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:55.138+0000 7ff0c77fe640 1 -- 172.21.15.114:0/3267801201 >> 172.21.15.114:0/3267801201 conn(0x7ff0c00fc4c0 msgr2=0x7ff0c0103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:55.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:55.138+0000 7ff0c77fe640 1 -- 172.21.15.114:0/3267801201 shutdown_connections 2024-09-20T19:42:55.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:55.138+0000 7ff0c77fe640 1 -- 172.21.15.114:0/3267801201 wait complete. 2024-09-20T19:42:55.609 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:55 smithi114 ceph-mon[28260]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-09-20T19:42:55.609 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:55 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:55.609 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:55 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:55.609 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:55 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/3267801201' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-20T19:42:55.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:55 smithi160 ceph-mon[35859]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-09-20T19:42:55.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:55 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:55.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:55 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:55.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:55 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/3267801201' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-20T19:42:55.946 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-20T19:42:55.947 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-20T19:42:55.947 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-09-20T19:42:55.947 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-20T19:42:55.947 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-20T19:42:55.947 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-20T19:42:55.947 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-20T19:42:55.947 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-20T19:42:55.947 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-20T19:42:55.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:42:55.947 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-09-20T19:42:55.963 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-20T19:42:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-20T19:42:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-09-20T19:42:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-09-20T19:42:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-09-20T19:42:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-09-20T19:42:55.964 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-20T19:42:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-09-20T19:42:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-09-20T19:42:55.964 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-20T19:42:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-09-20T19:42:55.964 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-09-20T19:42:55.965 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-20T19:42:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-09-20T19:42:55.965 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-20T19:42:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-09-20T19:42:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-20T19:42:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42348sec preferred_lft 42348sec 2024-09-20T19:42:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link noprefixroute 2024-09-20T19:42:55.965 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-09-20T19:42:55.966 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-20T19:42:55.966 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-20T19:42:55.966 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-09-20T19:42:55.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add name ceph-brx type bridge 2024-09-20T19:42:55.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr flush dev ceph-brx 2024-09-20T19:42:55.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set ceph-brx up 2024-09-20T19:42:55.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-20T19:42:55.966 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-09-20T19:42:56.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-20T19:42:56.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:42:56.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:42:56.214 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-20T19:42:56.239 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-09-20T19:42:56.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:42:56.241 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-09-20T19:42:56.325 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-09-20T19:42:56.325 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-20T19:42:56.325 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-20T19:42:56.325 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-20T19:42:56.325 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-20T19:42:56.326 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-20T19:42:56.326 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-09-20T19:42:56.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-20T19:42:56.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-20T19:42:56.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-20T19:42:56.326 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-09-20T19:42:56.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-20T19:42:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:42:56.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:42:56.750 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-09-20T19:42:56.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:42:56.766 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list-id 2024-09-20T19:42:56.821 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-20T19:42:56.822 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-09-20T19:42:56.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-20T19:42:56.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-20T19:42:56.822 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-09-20T19:42:56.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-20T19:42:56.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:42:56.939 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-20T19:42:56.939 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-20T19:42:56.939 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-09-20T19:42:56.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-20T19:42:56.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-20T19:42:56.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-20T19:42:56.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-20T19:42:56.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-20T19:42:56.940 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-09-20T19:42:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-20T19:42:57.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:42:57.101 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-20T19:42:57.101 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-09-20T19:42:57.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-09-20T19:42:57.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set dev brx.0 master ceph-brx 2024-09-20T19:42:57.101 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-09-20T19:42:57.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-20T19:42:57.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:42:57.181 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-20T19:42:57.181 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-20T19:42:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-20T19:42:57.198 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-20T19:42:57.198 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-20T19:42:57.254 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-20T19:42:57.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:42:57.254 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-20T19:42:57.395 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-20T19:42:57.395 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-20T19:42:57.396 INFO:teuthology.orchestra.run.smithi114.stdout:invalid new device string format 2024-09-20T19:42:57.605 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:57 smithi114 ceph-mon[28260]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:57.605 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:57 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:57.605 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:57 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:42:57.615 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-20T19:42:57.615 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-20T19:42:57.615 INFO:teuthology.orchestra.run.smithi114.stdout:invalid new device string format 2024-09-20T19:42:57.615 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.160:3300" 2024-09-20T19:42:57.615 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-20T19:42:57.619 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-20T19:42:57.620 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-20T19:42:57.620 INFO:teuthology.orchestra.run.smithi114.stdout:invalid new device string format 2024-09-20T19:42:57.620 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.160:3300" 2024-09-20T19:42:57.620 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-20T19:42:57.620 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:6789,172.21.15.160:6789" 2024-09-20T19:42:57.620 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.114:6789,172.21.15.160:6789:/ 2024-09-20T19:42:57.620 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=6a49c7bc-7787-11ef-baf5-efdc52797490" will pass to kernel 2024-09-20T19:42:57.621 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-20T19:42:57.621 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-09-20T19:42:57.621 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-20T19:42:57.621 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-09-20T19:42:57.621 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-20T19:42:57.623 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-20T19:42:57.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:42:57.623 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-20T19:42:57.651 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-20T19:42:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-20T19:42:57.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:42:57.751 INFO:teuthology.orchestra.run.smithi114.stdout:client24317 2024-09-20T19:42:57.751 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:42:57.751 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-20T19:42:57.768 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-20T19:42:57.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:57 smithi160 ceph-mon[35859]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:57.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:57 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:42:57.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:57 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:42:57.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-20T19:42:57.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:42:57.868 INFO:teuthology.orchestra.run.smithi114.stdout:client24317 2024-09-20T19:42:57.868 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-20T19:42:57.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-20T19:42:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:42:57.933 INFO:teuthology.orchestra.run.smithi114.stdout:client24317 2024-09-20T19:42:57.933 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490.client24317/status') 2024-09-20T19:42:57.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-20T19:42:58.002 INFO:teuthology.orchestra.run.smithi114.stdout:instance: client.24317 (3)172.21.15.114:0/1765762535 2024-09-20T19:42:58.002 INFO:teuthology.orchestra.run.smithi114.stdout:blocklisted: false 2024-09-20T19:42:58.002 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records in 2024-09-20T19:42:58.002 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records out 2024-09-20T19:42:58.002 INFO:teuthology.orchestra.run.smithi114.stderr:72 bytes copied, 2.4371e-05 s, 3.0 MB/s 2024-09-20T19:42:58.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:42:58.009 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-09-20T19:42:58.009 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-20T19:42:58.010 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-20T19:42:58.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph fs ls 2024-09-20T19:42:58.651 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:42:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.154+0000 7f43768f0640 1 -- 172.21.15.114:0/3225500647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4370106820 msgr2=0x7f4370106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.154+0000 7f43768f0640 1 --2- 172.21.15.114:0/3225500647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4370106820 0x7f4370106c00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4360009880 tx=0x7f436002a4a0 comp rx=0 tx=0).stop 2024-09-20T19:42:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.154+0000 7f43768f0640 1 -- 172.21.15.114:0/3225500647 shutdown_connections 2024-09-20T19:42:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.154+0000 7f43768f0640 1 --2- 172.21.15.114:0/3225500647 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4370100820 0x7f4370100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.154+0000 7f43768f0640 1 --2- 172.21.15.114:0/3225500647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4370106820 0x7f4370106c00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.154+0000 7f43768f0640 1 -- 172.21.15.114:0/3225500647 >> 172.21.15.114:0/3225500647 conn(0x7f43700fc520 msgr2=0x7f43700fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.154+0000 7f43768f0640 1 -- 172.21.15.114:0/3225500647 shutdown_connections 2024-09-20T19:42:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.155+0000 7f43768f0640 1 -- 172.21.15.114:0/3225500647 wait complete. 2024-09-20T19:42:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.155+0000 7f43768f0640 1 Processor -- start 2024-09-20T19:42:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.155+0000 7f43768f0640 1 -- start start 2024-09-20T19:42:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.155+0000 7f43768f0640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4370100820 0x7f437019d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.155+0000 7f43768f0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4370106820 0x7f437019dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.155+0000 7f43768f0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f437010dd50 con 0x7f4370106820 2024-09-20T19:42:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.155+0000 7f43768f0640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f437010dbd0 con 0x7f4370100820 2024-09-20T19:42:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.156+0000 7f436f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4370106820 0x7f437019dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.156+0000 7f436f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4370106820 0x7f437019dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55026/0 (socket says 172.21.15.114:55026) 2024-09-20T19:42:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.156+0000 7f436f7fe640 1 -- 172.21.15.114:0/2898903583 learned_addr learned my addr 172.21.15.114:0/2898903583 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:42:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.156+0000 7f436ffff640 1 --2- 172.21.15.114:0/2898903583 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4370100820 0x7f437019d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.156+0000 7f436f7fe640 1 -- 172.21.15.114:0/2898903583 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4370100820 msgr2=0x7f437019d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.156+0000 7f436f7fe640 1 --2- 172.21.15.114:0/2898903583 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4370100820 0x7f437019d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.156+0000 7f436f7fe640 1 -- 172.21.15.114:0/2898903583 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4370197840 con 0x7f4370106820 2024-09-20T19:42:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.156+0000 7f436f7fe640 1 --2- 172.21.15.114:0/2898903583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4370106820 0x7f437019dc00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f436400b420 tx=0x7f436400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.156+0000 7f436d7fa640 1 -- 172.21.15.114:0/2898903583 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f4364011630 con 0x7f4370106820 2024-09-20T19:42:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.156+0000 7f436d7fa640 1 -- 172.21.15.114:0/2898903583 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4364016070 con 0x7f4370106820 2024-09-20T19:42:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.156+0000 7f43768f0640 1 -- 172.21.15.114:0/2898903583 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4370197b30 con 0x7f4370106820 2024-09-20T19:42:59.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.156+0000 7f436d7fa640 1 -- 172.21.15.114:0/2898903583 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f4364010d40 con 0x7f4370106820 2024-09-20T19:42:59.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.156+0000 7f43768f0640 1 -- 172.21.15.114:0/2898903583 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43701980c0 con 0x7f4370106820 2024-09-20T19:42:59.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.157+0000 7f43768f0640 1 -- 172.21.15.114:0/2898903583 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4370101f60 con 0x7f4370106820 2024-09-20T19:42:59.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.158+0000 7f436d7fa640 1 -- 172.21.15.114:0/2898903583 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f43640117d0 con 0x7f4370106820 2024-09-20T19:42:59.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.158+0000 7f436d7fa640 1 --2- 172.21.15.114:0/2898903583 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f435806e2b0 0x7f4358070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:42:59.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.158+0000 7f436d7fa640 1 -- 172.21.15.114:0/2898903583 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f436408d9a0 con 0x7f4370106820 2024-09-20T19:42:59.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.158+0000 7f436ffff640 1 --2- 172.21.15.114:0/2898903583 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f435806e2b0 0x7f4358070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:42:59.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.159+0000 7f436ffff640 1 --2- 172.21.15.114:0/2898903583 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f435806e2b0 0x7f4358070770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f43600096c0 tx=0x7f4360031400 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:42:59.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.163+0000 7f436d7fa640 1 -- 172.21.15.114:0/2898903583 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f436405b300 con 0x7f4370106820 2024-09-20T19:42:59.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.395+0000 7f43768f0640 1 -- 172.21.15.114:0/2898903583 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4370109da0 con 0x7f4370106820 2024-09-20T19:42:59.396 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:42:59 smithi114 ceph-mon[28260]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:42:59.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.395+0000 7f436d7fa640 1 -- 172.21.15.114:0/2898903583 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) ==== 53+0+83 (secure 0 0 0) 0x7f436405efb0 con 0x7f4370106820 2024-09-20T19:42:59.397 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-09-20T19:42:59.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.397+0000 7f43768f0640 1 -- 172.21.15.114:0/2898903583 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f435806e2b0 msgr2=0x7f4358070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:59.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.397+0000 7f43768f0640 1 --2- 172.21.15.114:0/2898903583 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f435806e2b0 0x7f4358070770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f43600096c0 tx=0x7f4360031400 comp rx=0 tx=0).stop 2024-09-20T19:42:59.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.397+0000 7f43768f0640 1 -- 172.21.15.114:0/2898903583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4370106820 msgr2=0x7f437019dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:42:59.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.397+0000 7f43768f0640 1 --2- 172.21.15.114:0/2898903583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4370106820 0x7f437019dc00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f436400b420 tx=0x7f436400b8e0 comp rx=0 tx=0).stop 2024-09-20T19:42:59.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.397+0000 7f43768f0640 1 -- 172.21.15.114:0/2898903583 shutdown_connections 2024-09-20T19:42:59.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.397+0000 7f43768f0640 1 --2- 172.21.15.114:0/2898903583 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f435806e2b0 0x7f4358070770 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:59.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.397+0000 7f43768f0640 1 --2- 172.21.15.114:0/2898903583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4370106820 0x7f437019dc00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:59.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.397+0000 7f43768f0640 1 --2- 172.21.15.114:0/2898903583 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4370100820 0x7f437019d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:42:59.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.397+0000 7f43768f0640 1 -- 172.21.15.114:0/2898903583 >> 172.21.15.114:0/2898903583 conn(0x7f43700fc520 msgr2=0x7f437006a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:42:59.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.397+0000 7f43768f0640 1 -- 172.21.15.114:0/2898903583 shutdown_connections 2024-09-20T19:42:59.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:42:59.397+0000 7f43768f0640 1 -- 172.21.15.114:0/2898903583 wait complete. 2024-09-20T19:42:59.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:42:59 smithi160 ceph-mon[35859]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:00.127 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-20T19:43:00.127 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-20T19:43:00.127 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi160.front.sepia.ceph.com 2024-09-20T19:43:00.127 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-20T19:43:00.127 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-20T19:43:00.127 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-20T19:43:00.127 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-20T19:43:00.128 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-20T19:43:00.128 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-20T19:43:00.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:43:00.128 DEBUG:teuthology.orchestra.run.smithi160:> ip addr 2024-09-20T19:43:00.147 INFO:teuthology.orchestra.run.smithi160.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-20T19:43:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-20T19:43:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: inet 127.0.0.1/8 scope host lo 2024-09-20T19:43:00.147 INFO:teuthology.orchestra.run.smithi160.stdout: valid_lft forever preferred_lft forever 2024-09-20T19:43:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: inet6 ::1/128 scope host 2024-09-20T19:43:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: valid_lft forever preferred_lft forever 2024-09-20T19:43:00.148 INFO:teuthology.orchestra.run.smithi160.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-20T19:43:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: link/ether 0c:c4:7a:d9:82:80 brd ff:ff:ff:ff:ff:ff 2024-09-20T19:43:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: altname enp7s0f0 2024-09-20T19:43:00.148 INFO:teuthology.orchestra.run.smithi160.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-20T19:43:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: link/ether 0c:c4:7a:d9:82:81 brd ff:ff:ff:ff:ff:ff 2024-09-20T19:43:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: altname enp7s0f1 2024-09-20T19:43:00.148 INFO:teuthology.orchestra.run.smithi160.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-20T19:43:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: link/ether 0c:c4:7a:88:7f:5a brd ff:ff:ff:ff:ff:ff 2024-09-20T19:43:00.148 INFO:teuthology.orchestra.run.smithi160.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-20T19:43:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: link/ether 0c:c4:7a:88:7f:5b brd ff:ff:ff:ff:ff:ff 2024-09-20T19:43:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: inet 172.21.15.160/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-20T19:43:00.148 INFO:teuthology.orchestra.run.smithi160.stdout: valid_lft 42362sec preferred_lft 42362sec 2024-09-20T19:43:00.149 INFO:teuthology.orchestra.run.smithi160.stdout: inet6 fe80::ec4:7aff:fe88:7f5b/64 scope link noprefixroute 2024-09-20T19:43:00.149 INFO:teuthology.orchestra.run.smithi160.stdout: valid_lft forever preferred_lft forever 2024-09-20T19:43:00.149 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-20T19:43:00.149 DEBUG:teuthology.orchestra.run.smithi160:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-20T19:43:00.149 DEBUG:teuthology.orchestra.run.smithi160:> set -e 2024-09-20T19:43:00.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo ip link add name ceph-brx type bridge 2024-09-20T19:43:00.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo ip addr flush dev ceph-brx 2024-09-20T19:43:00.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo ip link set ceph-brx up 2024-09-20T19:43:00.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-20T19:43:00.149 DEBUG:teuthology.orchestra.run.smithi160:> ') 2024-09-20T19:43:00.243 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-20T19:43:00.372 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:43:00.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:43:00.379 DEBUG:teuthology.orchestra.run.smithi160:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-20T19:43:00.404 INFO:teuthology.orchestra.run.smithi160.stdout:1 2024-09-20T19:43:00.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:43:00.406 DEBUG:teuthology.orchestra.run.smithi160:> route 2024-09-20T19:43:00.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:00 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:43:00.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:00 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:43:00.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:00 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:00.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:00 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:00.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:00 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:43:00.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:00 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/2898903583' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-20T19:43:00.468 INFO:teuthology.orchestra.run.smithi160.stdout:Kernel IP routing table 2024-09-20T19:43:00.468 INFO:teuthology.orchestra.run.smithi160.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-20T19:43:00.468 INFO:teuthology.orchestra.run.smithi160.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-20T19:43:00.468 INFO:teuthology.orchestra.run.smithi160.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-20T19:43:00.468 INFO:teuthology.orchestra.run.smithi160.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-20T19:43:00.468 DEBUG:teuthology.orchestra.run.smithi160:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-20T19:43:00.468 DEBUG:teuthology.orchestra.run.smithi160:> set -e 2024-09-20T19:43:00.468 DEBUG:teuthology.orchestra.run.smithi160:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-20T19:43:00.468 DEBUG:teuthology.orchestra.run.smithi160:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-20T19:43:00.469 DEBUG:teuthology.orchestra.run.smithi160:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-20T19:43:00.469 DEBUG:teuthology.orchestra.run.smithi160:> ') 2024-09-20T19:43:00.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:00 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:43:00.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:00 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:43:00.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:00 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:00.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:00 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:00.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:00 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:43:00.530 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:00 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/2898903583' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-20T19:43:00.557 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-20T19:43:00.773 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:43:00.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:43:00.779 DEBUG:teuthology.orchestra.run.smithi160:> ip netns list 2024-09-20T19:43:00.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:43:00.835 DEBUG:teuthology.orchestra.run.smithi160:> ip netns list-id 2024-09-20T19:43:00.891 DEBUG:teuthology.orchestra.run.smithi160:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-20T19:43:00.891 DEBUG:teuthology.orchestra.run.smithi160:> set -e 2024-09-20T19:43:00.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-20T19:43:00.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-09-20T19:43:00.891 DEBUG:teuthology.orchestra.run.smithi160:> ') 2024-09-20T19:43:00.979 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-20T19:43:01.003 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:43:01.009 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-09-20T19:43:01.009 DEBUG:teuthology.orchestra.run.smithi160:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-20T19:43:01.009 DEBUG:teuthology.orchestra.run.smithi160:> set -e 2024-09-20T19:43:01.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-09-20T19:43:01.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-20T19:43:01.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-09-20T19:43:01.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-09-20T19:43:01.010 DEBUG:teuthology.orchestra.run.smithi160:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-09-20T19:43:01.010 DEBUG:teuthology.orchestra.run.smithi160:> ') 2024-09-20T19:43:01.058 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-20T19:43:01.125 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:43:01.131 DEBUG:teuthology.orchestra.run.smithi160:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-20T19:43:01.131 DEBUG:teuthology.orchestra.run.smithi160:> set -e 2024-09-20T19:43:01.132 DEBUG:teuthology.orchestra.run.smithi160:> sudo ip link set brx.0 up 2024-09-20T19:43:01.132 DEBUG:teuthology.orchestra.run.smithi160:> sudo ip link set dev brx.0 master ceph-brx 2024-09-20T19:43:01.132 DEBUG:teuthology.orchestra.run.smithi160:> ') 2024-09-20T19:43:01.180 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-20T19:43:01.203 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:43:01.209 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-20T19:43:01.209 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-20T19:43:01.224 INFO:teuthology.orchestra.run.smithi160.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-20T19:43:01.224 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-20T19:43:01.224 DEBUG:teuthology.orchestra.run.smithi160:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-20T19:43:01.279 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-20T19:43:01.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:43:01.279 DEBUG:teuthology.orchestra.run.smithi160:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-20T19:43:01.374 INFO:teuthology.orchestra.run.smithi160.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-20T19:43:01.374 INFO:teuthology.orchestra.run.smithi160.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-20T19:43:01.374 INFO:teuthology.orchestra.run.smithi160.stdout:invalid new device string format 2024-09-20T19:43:01.500 INFO:teuthology.orchestra.run.smithi160.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-20T19:43:01.500 INFO:teuthology.orchestra.run.smithi160.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-20T19:43:01.500 INFO:teuthology.orchestra.run.smithi160.stdout:invalid new device string format 2024-09-20T19:43:01.500 INFO:teuthology.orchestra.run.smithi160.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.160:3300" 2024-09-20T19:43:01.500 INFO:teuthology.orchestra.run.smithi160.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-20T19:43:01.505 INFO:teuthology.orchestra.run.smithi160.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-20T19:43:01.505 INFO:teuthology.orchestra.run.smithi160.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-20T19:43:01.505 INFO:teuthology.orchestra.run.smithi160.stdout:invalid new device string format 2024-09-20T19:43:01.505 INFO:teuthology.orchestra.run.smithi160.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.160:3300" 2024-09-20T19:43:01.505 INFO:teuthology.orchestra.run.smithi160.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-20T19:43:01.505 INFO:teuthology.orchestra.run.smithi160.stdout:mount.ceph: resolved to: "172.21.15.114:6789,172.21.15.160:6789" 2024-09-20T19:43:01.505 INFO:teuthology.orchestra.run.smithi160.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.114:6789,172.21.15.160:6789:/ 2024-09-20T19:43:01.505 INFO:teuthology.orchestra.run.smithi160.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=6a49c7bc-7787-11ef-baf5-efdc52797490" will pass to kernel 2024-09-20T19:43:01.506 INFO:teuthology.orchestra.run.smithi160.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-20T19:43:01.506 INFO:teuthology.orchestra.run.smithi160.stdout: You just mounted a file system that supports labels which does not 2024-09-20T19:43:01.506 INFO:teuthology.orchestra.run.smithi160.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-20T19:43:01.506 INFO:teuthology.orchestra.run.smithi160.stdout: applications will generate AVC messages and not be allowed access to 2024-09-20T19:43:01.506 INFO:teuthology.orchestra.run.smithi160.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-20T19:43:01.508 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-20T19:43:01.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T19:43:01.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-20T19:43:01.532 DEBUG:teuthology.orchestra.run.smithi160:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-20T19:43:01.621 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-20T19:43:01.623 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:43:01.629 INFO:teuthology.orchestra.run.smithi160.stdout:client24321 2024-09-20T19:43:01.630 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-20T19:43:01.630 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-20T19:43:01.647 DEBUG:teuthology.orchestra.run.smithi160:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-20T19:43:01.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:01 smithi114 ceph-mon[28260]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-09-20T19:43:01.735 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-20T19:43:01.737 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:43:01.743 INFO:teuthology.orchestra.run.smithi160.stdout:client24321 2024-09-20T19:43:01.743 DEBUG:teuthology.orchestra.run.smithi160:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-20T19:43:01.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:01 smithi160 ceph-mon[35859]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-09-20T19:43:01.791 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-20T19:43:01.796 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:43:01.802 INFO:teuthology.orchestra.run.smithi160.stdout:client24321 2024-09-20T19:43:01.802 DEBUG:teuthology.orchestra.run.smithi160:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490.client24321/status') 2024-09-20T19:43:01.890 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-20T19:43:01.900 INFO:teuthology.orchestra.run.smithi160.stdout:instance: client.24321 (3)192.168.0.1:0/3516892091 2024-09-20T19:43:01.900 INFO:teuthology.orchestra.run.smithi160.stdout:blocklisted: false 2024-09-20T19:43:01.901 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records in 2024-09-20T19:43:01.901 INFO:teuthology.orchestra.run.smithi160.stderr:0+1 records out 2024-09-20T19:43:01.901 INFO:teuthology.orchestra.run.smithi160.stderr:70 bytes copied, 2.1322e-05 s, 3.3 MB/s 2024-09-20T19:43:01.901 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-20T19:43:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-20T19:43:01.908 INFO:teuthology.run_tasks:Running task print... 2024-09-20T19:43:01.918 INFO:teuthology.task.print:**** done client 2024-09-20T19:43:01.918 INFO:teuthology.run_tasks:Running task parallel... 2024-09-20T19:43:01.940 INFO:teuthology.task.parallel:starting parallel... 2024-09-20T19:43:01.941 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-20T19:43:01.941 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-09-20T19:43:01.942 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-09-20T19:43:01.942 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-20T19:43:01.943 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-20T19:43:01.943 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-20T19:43:01.947 INFO:tasks.workunit:Pulling workunits from ref e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:43:01.947 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-20T19:43:01.947 INFO:tasks.workunit:timeout=3h 2024-09-20T19:43:01.947 INFO:tasks.workunit:cleanup=True 2024-09-20T19:43:01.948 DEBUG:teuthology.orchestra.run.smithi114:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-20T19:43:01.968 INFO:teuthology.orchestra.run.smithi114.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-20T19:43:01.968 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-09-20T19:43:01.968 INFO:teuthology.orchestra.run.smithi114.stdout:Device: a5h/165d Inode: 1 Links: 2 2024-09-20T19:43:01.968 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:43:01.968 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-09-20T19:43:01.968 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-09-20T19:43:01.968 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-20 19:42:37.665150992 +0000 2024-09-20T19:43:01.968 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-20 19:42:57.647146838 +0000 2024-09-20T19:43:01.969 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-20 19:42:37.665150992 +0000 2024-09-20T19:43:01.969 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-20T19:43:01.969 DEBUG:teuthology.orchestra.run.smithi114:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-20T19:43:02.040 DEBUG:teuthology.orchestra.run.smithi160:> stat -- /home/ubuntu/cephtest/mnt.1 2024-09-20T19:43:02.057 INFO:teuthology.orchestra.run.smithi160.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-09-20T19:43:02.057 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-09-20T19:43:02.057 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 81h/129d Inode: 1 Links: 3 2024-09-20T19:43:02.057 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T19:43:02.057 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-09-20T19:43:02.057 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-09-20T19:43:02.057 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-20 19:43:02.033049194 +0000 2024-09-20T19:43:02.057 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-20 19:43:02.033049194 +0000 2024-09-20T19:43:02.058 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-20 19:42:37.665150992 +0000 2024-09-20T19:43:02.058 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-09-20T19:43:02.058 DEBUG:teuthology.orchestra.run.smithi160:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-09-20T19:43:02.126 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:43:02.127 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:43:02.143 INFO:tasks.workunit.client.0.smithi114.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-20T19:43:02.182 INFO:tasks.workunit.client.1.smithi160.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-09-20T19:43:02.219 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:43:02.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.680+0000 7fe771834640 1 -- 172.21.15.114:0/963584914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe76c100880 msgr2=0x7fe76c100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:02.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.680+0000 7fe771834640 1 --2- 172.21.15.114:0/963584914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe76c100880 0x7fe76c100c80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe764009c20 tx=0x7fe76402a970 comp rx=0 tx=0).stop 2024-09-20T19:43:02.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.680+0000 7fe771834640 1 -- 172.21.15.114:0/963584914 shutdown_connections 2024-09-20T19:43:02.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.680+0000 7fe771834640 1 --2- 172.21.15.114:0/963584914 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe76c101a80 0x7fe76c101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:02.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.680+0000 7fe771834640 1 --2- 172.21.15.114:0/963584914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe76c100880 0x7fe76c100c80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:02.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.680+0000 7fe771834640 1 -- 172.21.15.114:0/963584914 >> 172.21.15.114:0/963584914 conn(0x7fe76c0fc010 msgr2=0x7fe76c0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:02.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.680+0000 7fe771834640 1 -- 172.21.15.114:0/963584914 shutdown_connections 2024-09-20T19:43:02.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.680+0000 7fe771834640 1 -- 172.21.15.114:0/963584914 wait complete. 2024-09-20T19:43:02.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.681+0000 7fe771834640 1 Processor -- start 2024-09-20T19:43:02.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.681+0000 7fe771834640 1 -- start start 2024-09-20T19:43:02.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.681+0000 7fe771834640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe76c100880 0x7fe76c06adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:02.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.681+0000 7fe771834640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe76c101a80 0x7fe76c19c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:02.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.681+0000 7fe771834640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe76c109720 con 0x7fe76c101a80 2024-09-20T19:43:02.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.681+0000 7fe771834640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe76c1095a0 con 0x7fe76c100880 2024-09-20T19:43:02.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.681+0000 7fe76a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe76c101a80 0x7fe76c19c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:02.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.681+0000 7fe76a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe76c101a80 0x7fe76c19c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55042/0 (socket says 172.21.15.114:55042) 2024-09-20T19:43:02.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.681+0000 7fe76a7fc640 1 -- 172.21.15.114:0/3398122713 learned_addr learned my addr 172.21.15.114:0/3398122713 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:43:02.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.681+0000 7fe76affd640 1 --2- 172.21.15.114:0/3398122713 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe76c100880 0x7fe76c06adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:02.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.682+0000 7fe76a7fc640 1 -- 172.21.15.114:0/3398122713 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe76c100880 msgr2=0x7fe76c06adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:02.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.682+0000 7fe76a7fc640 1 --2- 172.21.15.114:0/3398122713 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe76c100880 0x7fe76c06adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:02.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.682+0000 7fe76a7fc640 1 -- 172.21.15.114:0/3398122713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe76c19c8c0 con 0x7fe76c101a80 2024-09-20T19:43:02.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.682+0000 7fe76a7fc640 1 --2- 172.21.15.114:0/3398122713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe76c101a80 0x7fe76c19c300 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe75400b420 tx=0x7fe75400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:02.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.682+0000 7fe770832640 1 -- 172.21.15.114:0/3398122713 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe754011630 con 0x7fe76c101a80 2024-09-20T19:43:02.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.682+0000 7fe770832640 1 -- 172.21.15.114:0/3398122713 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe754016070 con 0x7fe76c101a80 2024-09-20T19:43:02.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.682+0000 7fe771834640 1 -- 172.21.15.114:0/3398122713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe76c19cbb0 con 0x7fe76c101a80 2024-09-20T19:43:02.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.682+0000 7fe770832640 1 -- 172.21.15.114:0/3398122713 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe754010d30 con 0x7fe76c101a80 2024-09-20T19:43:02.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.682+0000 7fe771834640 1 -- 172.21.15.114:0/3398122713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe76c19d1c0 con 0x7fe76c101a80 2024-09-20T19:43:02.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.683+0000 7fe771834640 1 -- 172.21.15.114:0/3398122713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe76c100d00 con 0x7fe76c101a80 2024-09-20T19:43:02.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.683+0000 7fe770832640 1 -- 172.21.15.114:0/3398122713 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fe75401b460 con 0x7fe76c101a80 2024-09-20T19:43:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.684+0000 7fe770832640 1 --2- 172.21.15.114:0/3398122713 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fe75806e2b0 0x7fe758070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.684+0000 7fe770832640 1 -- 172.21.15.114:0/3398122713 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7fe75408d9a0 con 0x7fe76c101a80 2024-09-20T19:43:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.684+0000 7fe76affd640 1 --2- 172.21.15.114:0/3398122713 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fe75806e2b0 0x7fe758070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:02.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.685+0000 7fe76affd640 1 --2- 172.21.15.114:0/3398122713 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fe75806e2b0 0x7fe758070770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe764000f80 tx=0x7fe76403c040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:02.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.688+0000 7fe770832640 1 -- 172.21.15.114:0/3398122713 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fe75405b270 con 0x7fe76c101a80 2024-09-20T19:43:02.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.854+0000 7fe771834640 1 -- 172.21.15.114:0/3398122713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) -- 0x7fe76c19db60 con 0x7fe76c101a80 2024-09-20T19:43:02.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.855+0000 7fe770832640 1 -- 172.21.15.114:0/3398122713 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v42)=0 v42) ==== 155+0+0 (secure 0 0 0) 0x7fe75405ef20 con 0x7fe76c101a80 2024-09-20T19:43:02.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.857+0000 7fe771834640 1 -- 172.21.15.114:0/3398122713 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fe75806e2b0 msgr2=0x7fe758070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:02.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.857+0000 7fe771834640 1 --2- 172.21.15.114:0/3398122713 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fe75806e2b0 0x7fe758070770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe764000f80 tx=0x7fe76403c040 comp rx=0 tx=0).stop 2024-09-20T19:43:02.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.857+0000 7fe771834640 1 -- 172.21.15.114:0/3398122713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe76c101a80 msgr2=0x7fe76c19c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:02.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.857+0000 7fe771834640 1 --2- 172.21.15.114:0/3398122713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe76c101a80 0x7fe76c19c300 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe75400b420 tx=0x7fe75400b8e0 comp rx=0 tx=0).stop 2024-09-20T19:43:02.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.857+0000 7fe771834640 1 -- 172.21.15.114:0/3398122713 shutdown_connections 2024-09-20T19:43:02.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.857+0000 7fe771834640 1 --2- 172.21.15.114:0/3398122713 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fe75806e2b0 0x7fe758070770 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:02.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.857+0000 7fe771834640 1 --2- 172.21.15.114:0/3398122713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe76c101a80 0x7fe76c19c300 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:02.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.857+0000 7fe771834640 1 --2- 172.21.15.114:0/3398122713 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe76c100880 0x7fe76c06adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:02.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.857+0000 7fe771834640 1 -- 172.21.15.114:0/3398122713 >> 172.21.15.114:0/3398122713 conn(0x7fe76c0fc010 msgr2=0x7fe76c0fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:02.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.857+0000 7fe771834640 1 -- 172.21.15.114:0/3398122713 shutdown_connections 2024-09-20T19:43:02.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:02.857+0000 7fe771834640 1 -- 172.21.15.114:0/3398122713 wait complete. 2024-09-20T19:43:03.565 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:03 smithi114 ceph-mon[28260]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-09-20T19:43:03.594 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-20T19:43:03.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:03 smithi160 ceph-mon[35859]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-09-20T19:43:03.882 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:43:04.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.318+0000 7f5d370de640 1 -- 172.21.15.114:0/2047124956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d30102490 msgr2=0x7f5d30104880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:04.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.318+0000 7f5d370de640 1 --2- 172.21.15.114:0/2047124956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d30102490 0x7f5d30104880 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009880 tx=0x7f5d2002a4a0 comp rx=0 tx=0).stop 2024-09-20T19:43:04.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.318+0000 7f5d370de640 1 -- 172.21.15.114:0/2047124956 shutdown_connections 2024-09-20T19:43:04.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.318+0000 7f5d370de640 1 --2- 172.21.15.114:0/2047124956 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5d30104dc0 0x7f5d301071b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:04.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.318+0000 7f5d370de640 1 --2- 172.21.15.114:0/2047124956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d30102490 0x7f5d30104880 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:04.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.318+0000 7f5d370de640 1 -- 172.21.15.114:0/2047124956 >> 172.21.15.114:0/2047124956 conn(0x7f5d300fbff0 msgr2=0x7f5d300fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:04.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.318+0000 7f5d370de640 1 -- 172.21.15.114:0/2047124956 shutdown_connections 2024-09-20T19:43:04.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.318+0000 7f5d370de640 1 -- 172.21.15.114:0/2047124956 wait complete. 2024-09-20T19:43:04.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.319+0000 7f5d370de640 1 Processor -- start 2024-09-20T19:43:04.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.319+0000 7f5d370de640 1 -- start start 2024-09-20T19:43:04.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.319+0000 7f5d370de640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5d30102490 0x7f5d301975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:04.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.319+0000 7f5d370de640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d30104dc0 0x7f5d30197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:04.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.319+0000 7f5d370de640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3010a800 con 0x7f5d30104dc0 2024-09-20T19:43:04.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.319+0000 7f5d370de640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3010a680 con 0x7f5d30102490 2024-09-20T19:43:04.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.319+0000 7f5d2ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d30104dc0 0x7f5d30197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:04.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.319+0000 7f5d2ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d30104dc0 0x7f5d30197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55070/0 (socket says 172.21.15.114:55070) 2024-09-20T19:43:04.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.319+0000 7f5d2ffff640 1 -- 172.21.15.114:0/3093799330 learned_addr learned my addr 172.21.15.114:0/3093799330 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:43:04.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.319+0000 7f5d2ffff640 1 -- 172.21.15.114:0/3093799330 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5d30102490 msgr2=0x7f5d301975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:04.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.319+0000 7f5d2ffff640 1 --2- 172.21.15.114:0/3093799330 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5d30102490 0x7f5d301975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:04.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.319+0000 7f5d2ffff640 1 -- 172.21.15.114:0/3093799330 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d301981a0 con 0x7f5d30104dc0 2024-09-20T19:43:04.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.320+0000 7f5d2ffff640 1 --2- 172.21.15.114:0/3093799330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d30104dc0 0x7f5d30197b20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400b3d0 tx=0x7f5d2400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:04.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.320+0000 7f5d2dffb640 1 -- 172.21.15.114:0/3093799330 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5d24015020 con 0x7f5d30104dc0 2024-09-20T19:43:04.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.320+0000 7f5d2dffb640 1 -- 172.21.15.114:0/3093799330 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5d24011af0 con 0x7f5d30104dc0 2024-09-20T19:43:04.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.320+0000 7f5d370de640 1 -- 172.21.15.114:0/3093799330 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d3019cbe0 con 0x7f5d30104dc0 2024-09-20T19:43:04.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.320+0000 7f5d2dffb640 1 -- 172.21.15.114:0/3093799330 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5d24010d30 con 0x7f5d30104dc0 2024-09-20T19:43:04.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.320+0000 7f5d370de640 1 -- 172.21.15.114:0/3093799330 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d3019d170 con 0x7f5d30104dc0 2024-09-20T19:43:04.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.321+0000 7f5d370de640 1 -- 172.21.15.114:0/3093799330 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d30102870 con 0x7f5d30104dc0 2024-09-20T19:43:04.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.321+0000 7f5d2dffb640 1 -- 172.21.15.114:0/3093799330 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f5d24018020 con 0x7f5d30104dc0 2024-09-20T19:43:04.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.322+0000 7f5d2dffb640 1 --2- 172.21.15.114:0/3093799330 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5d1806e2b0 0x7f5d18070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:04.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.322+0000 7f5d2dffb640 1 -- 172.21.15.114:0/3093799330 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f5d2408dcc0 con 0x7f5d30104dc0 2024-09-20T19:43:04.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.322+0000 7f5d34e7a640 1 --2- 172.21.15.114:0/3093799330 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5d1806e2b0 0x7f5d18070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:04.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.322+0000 7f5d34e7a640 1 --2- 172.21.15.114:0/3093799330 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5d1806e2b0 0x7f5d18070770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5d200096c0 tx=0x7f5d20000bc0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:04.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.326+0000 7f5d2dffb640 1 -- 172.21.15.114:0/3093799330 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f5d2401f020 con 0x7f5d30104dc0 2024-09-20T19:43:04.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.493+0000 7f5d370de640 1 -- 172.21.15.114:0/3093799330 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) -- 0x7f5d3019db40 con 0x7f5d30104dc0 2024-09-20T19:43:04.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.494+0000 7f5d2dffb640 1 -- 172.21.15.114:0/3093799330 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v42)=0 v42) ==== 163+0+0 (secure 0 0 0) 0x7f5d2405b5c0 con 0x7f5d30104dc0 2024-09-20T19:43:04.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.496+0000 7f5d370de640 1 -- 172.21.15.114:0/3093799330 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5d1806e2b0 msgr2=0x7f5d18070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:04.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.496+0000 7f5d370de640 1 --2- 172.21.15.114:0/3093799330 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5d1806e2b0 0x7f5d18070770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5d200096c0 tx=0x7f5d20000bc0 comp rx=0 tx=0).stop 2024-09-20T19:43:04.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.496+0000 7f5d370de640 1 -- 172.21.15.114:0/3093799330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d30104dc0 msgr2=0x7f5d30197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:04.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.496+0000 7f5d370de640 1 --2- 172.21.15.114:0/3093799330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d30104dc0 0x7f5d30197b20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400b3d0 tx=0x7f5d2400b890 comp rx=0 tx=0).stop 2024-09-20T19:43:04.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.496+0000 7f5d370de640 1 -- 172.21.15.114:0/3093799330 shutdown_connections 2024-09-20T19:43:04.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.496+0000 7f5d370de640 1 --2- 172.21.15.114:0/3093799330 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f5d1806e2b0 0x7f5d18070770 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:04.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.496+0000 7f5d370de640 1 --2- 172.21.15.114:0/3093799330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d30104dc0 0x7f5d30197b20 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:04.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.496+0000 7f5d370de640 1 --2- 172.21.15.114:0/3093799330 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5d30102490 0x7f5d301975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:04.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.496+0000 7f5d370de640 1 -- 172.21.15.114:0/3093799330 >> 172.21.15.114:0/3093799330 conn(0x7f5d300fbff0 msgr2=0x7f5d300fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:04.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.496+0000 7f5d370de640 1 -- 172.21.15.114:0/3093799330 shutdown_connections 2024-09-20T19:43:04.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:04.496+0000 7f5d370de640 1 -- 172.21.15.114:0/3093799330 wait complete. 2024-09-20T19:43:05.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-20T19:43:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:43:05.634 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:05 smithi114 ceph-mon[28260]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-09-20T19:43:05.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:05 smithi160 ceph-mon[35859]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-09-20T19:43:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.871+0000 7f7e238c3640 1 -- 172.21.15.114:0/2989208779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e1c102430 msgr2=0x7f7e1c104820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.871+0000 7f7e238c3640 1 --2- 172.21.15.114:0/2989208779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e1c102430 0x7f7e1c104820 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7e18009880 tx=0x7f7e1802a4a0 comp rx=0 tx=0).stop 2024-09-20T19:43:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.871+0000 7f7e238c3640 1 -- 172.21.15.114:0/2989208779 shutdown_connections 2024-09-20T19:43:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.871+0000 7f7e238c3640 1 --2- 172.21.15.114:0/2989208779 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7e1c104d60 0x7f7e1c107150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.871+0000 7f7e238c3640 1 --2- 172.21.15.114:0/2989208779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e1c102430 0x7f7e1c104820 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.871+0000 7f7e238c3640 1 -- 172.21.15.114:0/2989208779 >> 172.21.15.114:0/2989208779 conn(0x7f7e1c0fbff0 msgr2=0x7f7e1c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.871+0000 7f7e238c3640 1 -- 172.21.15.114:0/2989208779 shutdown_connections 2024-09-20T19:43:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.871+0000 7f7e238c3640 1 -- 172.21.15.114:0/2989208779 wait complete. 2024-09-20T19:43:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.872+0000 7f7e238c3640 1 Processor -- start 2024-09-20T19:43:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.872+0000 7f7e238c3640 1 -- start start 2024-09-20T19:43:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.872+0000 7f7e238c3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e1c102430 0x7f7e1c1974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.872+0000 7f7e238c3640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7e1c104d60 0x7f7e1c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.872+0000 7f7e238c3640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c10a7a0 con 0x7f7e1c102430 2024-09-20T19:43:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.872+0000 7f7e238c3640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c10a620 con 0x7f7e1c104d60 2024-09-20T19:43:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.872+0000 7f7e2165f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e1c102430 0x7f7e1c1974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.872+0000 7f7e2165f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e1c102430 0x7f7e1c1974f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46582/0 (socket says 172.21.15.114:46582) 2024-09-20T19:43:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.872+0000 7f7e2165f640 1 -- 172.21.15.114:0/2642343432 learned_addr learned my addr 172.21.15.114:0/2642343432 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:43:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.872+0000 7f7e20e5e640 1 --2- 172.21.15.114:0/2642343432 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7e1c104d60 0x7f7e1c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:05.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.872+0000 7f7e2165f640 1 -- 172.21.15.114:0/2642343432 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7e1c104d60 msgr2=0x7f7e1c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:05.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.872+0000 7f7e2165f640 1 --2- 172.21.15.114:0/2642343432 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7e1c104d60 0x7f7e1c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:05.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.872+0000 7f7e2165f640 1 -- 172.21.15.114:0/2642343432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1c1980b0 con 0x7f7e1c102430 2024-09-20T19:43:05.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.873+0000 7f7e2165f640 1 --2- 172.21.15.114:0/2642343432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e1c102430 0x7f7e1c1974f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7e18006010 tx=0x7f7e180048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:05.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.873+0000 7f7e127fc640 1 -- 172.21.15.114:0/2642343432 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f7e1803f070 con 0x7f7e1c102430 2024-09-20T19:43:05.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.873+0000 7f7e238c3640 1 -- 172.21.15.114:0/2642343432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e1c19caf0 con 0x7f7e1c102430 2024-09-20T19:43:05.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.873+0000 7f7e127fc640 1 -- 172.21.15.114:0/2642343432 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7e180044d0 con 0x7f7e1c102430 2024-09-20T19:43:05.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.873+0000 7f7e127fc640 1 -- 172.21.15.114:0/2642343432 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f7e18035030 con 0x7f7e1c102430 2024-09-20T19:43:05.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.873+0000 7f7e238c3640 1 -- 172.21.15.114:0/2642343432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e1c19d080 con 0x7f7e1c102430 2024-09-20T19:43:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.874+0000 7f7e238c3640 1 -- 172.21.15.114:0/2642343432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e1c102810 con 0x7f7e1c102430 2024-09-20T19:43:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.875+0000 7f7e127fc640 1 -- 172.21.15.114:0/2642343432 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f7e18042070 con 0x7f7e1c102430 2024-09-20T19:43:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.875+0000 7f7e127fc640 1 --2- 172.21.15.114:0/2642343432 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7e0806e2b0 0x7f7e08070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.876+0000 7f7e127fc640 1 -- 172.21.15.114:0/2642343432 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f7e180aee70 con 0x7f7e1c102430 2024-09-20T19:43:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.876+0000 7f7e20e5e640 1 --2- 172.21.15.114:0/2642343432 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7e0806e2b0 0x7f7e08070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.876+0000 7f7e20e5e640 1 --2- 172.21.15.114:0/2642343432 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7e0806e2b0 0x7f7e08070770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c1989a0 tx=0x7f7e0c008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:05.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:05.880+0000 7f7e127fc640 1 -- 172.21.15.114:0/2642343432 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f7e1807c7d0 con 0x7f7e1c102430 2024-09-20T19:43:06.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:06.051+0000 7f7e238c3640 1 -- 172.21.15.114:0/2642343432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) -- 0x7f7e1c19dae0 con 0x7f7e1c102430 2024-09-20T19:43:06.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:06.052+0000 7f7e127fc640 1 -- 172.21.15.114:0/2642343432 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v42)=0 v42) ==== 135+0+0 (secure 0 0 0) 0x7f7e18080480 con 0x7f7e1c102430 2024-09-20T19:43:06.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:06.054+0000 7f7e238c3640 1 -- 172.21.15.114:0/2642343432 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7e0806e2b0 msgr2=0x7f7e08070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:06.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:06.054+0000 7f7e238c3640 1 --2- 172.21.15.114:0/2642343432 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7e0806e2b0 0x7f7e08070770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c1989a0 tx=0x7f7e0c008040 comp rx=0 tx=0).stop 2024-09-20T19:43:06.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:06.054+0000 7f7e238c3640 1 -- 172.21.15.114:0/2642343432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e1c102430 msgr2=0x7f7e1c1974f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:06.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:06.054+0000 7f7e238c3640 1 --2- 172.21.15.114:0/2642343432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e1c102430 0x7f7e1c1974f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7e18006010 tx=0x7f7e180048f0 comp rx=0 tx=0).stop 2024-09-20T19:43:06.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:06.054+0000 7f7e238c3640 1 -- 172.21.15.114:0/2642343432 shutdown_connections 2024-09-20T19:43:06.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:06.054+0000 7f7e238c3640 1 --2- 172.21.15.114:0/2642343432 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f7e0806e2b0 0x7f7e08070770 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:06.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:06.054+0000 7f7e238c3640 1 --2- 172.21.15.114:0/2642343432 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7e1c104d60 0x7f7e1c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:06.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:06.054+0000 7f7e238c3640 1 --2- 172.21.15.114:0/2642343432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e1c102430 0x7f7e1c1974f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:06.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:06.054+0000 7f7e238c3640 1 -- 172.21.15.114:0/2642343432 >> 172.21.15.114:0/2642343432 conn(0x7f7e1c0fbff0 msgr2=0x7f7e1c0fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:06.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:06.054+0000 7f7e238c3640 1 -- 172.21.15.114:0/2642343432 shutdown_connections 2024-09-20T19:43:06.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:06.054+0000 7f7e238c3640 1 -- 172.21.15.114:0/2642343432 wait complete. 2024-09-20T19:43:06.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-09-20T19:43:06.905 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:43:07.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.395+0000 7fe73663e640 1 -- 172.21.15.114:0/2266229591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe730100880 msgr2=0x7fe730100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:07.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.395+0000 7fe73663e640 1 --2- 172.21.15.114:0/2266229591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe730100880 0x7fe730100c80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe720009880 tx=0x7fe72002a4a0 comp rx=0 tx=0).stop 2024-09-20T19:43:07.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.395+0000 7fe73663e640 1 -- 172.21.15.114:0/2266229591 shutdown_connections 2024-09-20T19:43:07.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.395+0000 7fe73663e640 1 --2- 172.21.15.114:0/2266229591 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe730101a80 0x7fe730101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:07.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.395+0000 7fe73663e640 1 --2- 172.21.15.114:0/2266229591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe730100880 0x7fe730100c80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:07.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.395+0000 7fe73663e640 1 -- 172.21.15.114:0/2266229591 >> 172.21.15.114:0/2266229591 conn(0x7fe7300fc010 msgr2=0x7fe7300fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:07.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.395+0000 7fe73663e640 1 -- 172.21.15.114:0/2266229591 shutdown_connections 2024-09-20T19:43:07.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.395+0000 7fe73663e640 1 -- 172.21.15.114:0/2266229591 wait complete. 2024-09-20T19:43:07.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.396+0000 7fe73663e640 1 Processor -- start 2024-09-20T19:43:07.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.396+0000 7fe73663e640 1 -- start start 2024-09-20T19:43:07.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe73663e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe730100880 0x7fe7301963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:07.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe73663e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe730101a80 0x7fe730196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:07.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe73663e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe7301096c0 con 0x7fe730100880 2024-09-20T19:43:07.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe73663e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fe730109540 con 0x7fe730101a80 2024-09-20T19:43:07.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe72ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe730100880 0x7fe7301963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:07.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe72ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe730100880 0x7fe7301963e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46600/0 (socket says 172.21.15.114:46600) 2024-09-20T19:43:07.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe72ffff640 1 -- 172.21.15.114:0/4265409584 learned_addr learned my addr 172.21.15.114:0/4265409584 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:43:07.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe72f7fe640 1 --2- 172.21.15.114:0/4265409584 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe730101a80 0x7fe730196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:07.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe72ffff640 1 -- 172.21.15.114:0/4265409584 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe730101a80 msgr2=0x7fe730196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:07.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe72ffff640 1 --2- 172.21.15.114:0/4265409584 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe730101a80 0x7fe730196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:07.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe72ffff640 1 -- 172.21.15.114:0/4265409584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe730196fa0 con 0x7fe730100880 2024-09-20T19:43:07.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe72ffff640 1 --2- 172.21.15.114:0/4265409584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe730100880 0x7fe7301963e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe720006010 tx=0x7fe7200048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:07.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe72d7fa640 1 -- 172.21.15.114:0/4265409584 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe72003f070 con 0x7fe730100880 2024-09-20T19:43:07.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe72d7fa640 1 -- 172.21.15.114:0/4265409584 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe7200044d0 con 0x7fe730100880 2024-09-20T19:43:07.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe73663e640 1 -- 172.21.15.114:0/4265409584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe73019b980 con 0x7fe730100880 2024-09-20T19:43:07.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe72d7fa640 1 -- 172.21.15.114:0/4265409584 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe720035030 con 0x7fe730100880 2024-09-20T19:43:07.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.397+0000 7fe73663e640 1 -- 172.21.15.114:0/4265409584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe73019bf10 con 0x7fe730100880 2024-09-20T19:43:07.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.398+0000 7fe73663e640 1 -- 172.21.15.114:0/4265409584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe730100c80 con 0x7fe730100880 2024-09-20T19:43:07.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.399+0000 7fe72d7fa640 1 -- 172.21.15.114:0/4265409584 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fe720042070 con 0x7fe730100880 2024-09-20T19:43:07.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.399+0000 7fe72d7fa640 1 --2- 172.21.15.114:0/4265409584 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fe71806e310 0x7fe7180707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:07.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.399+0000 7fe72f7fe640 1 --2- 172.21.15.114:0/4265409584 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fe71806e310 0x7fe7180707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:07.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.399+0000 7fe72d7fa640 1 -- 172.21.15.114:0/4265409584 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7fe7200aed50 con 0x7fe730100880 2024-09-20T19:43:07.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.400+0000 7fe72f7fe640 1 --2- 172.21.15.114:0/4265409584 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fe71806e310 0x7fe7180707d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe724005fd0 tx=0x7fe724005e20 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:07.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.404+0000 7fe72d7fa640 1 -- 172.21.15.114:0/4265409584 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fe72007c730 con 0x7fe730100880 2024-09-20T19:43:07.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:07.589+0000 7fe73663e640 1 -- 172.21.15.114:0/4265409584 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) -- 0x7fe7300626e0 con 0x7fe71806e310 2024-09-20T19:43:07.592 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:07 smithi114 ceph-mon[28260]: pgmap v189: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-09-20T19:43:07.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:07 smithi160 ceph-mon[35859]: pgmap v189: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-09-20T19:43:09.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:09 smithi114 ceph-mon[28260]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:09.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:09 smithi114 ceph-mon[28260]: pgmap v190: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-09-20T19:43:09.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:09 smithi160 ceph-mon[35859]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:09.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:09 smithi160 ceph-mon[35859]: pgmap v190: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-09-20T19:43:11.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:11 smithi114 ceph-mon[28260]: pgmap v191: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-09-20T19:43:11.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:11 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:43:11.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:11 smithi160 ceph-mon[35859]: pgmap v191: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-09-20T19:43:11.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:11 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:43:13.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:13 smithi114 ceph-mon[28260]: pgmap v192: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-09-20T19:43:13.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:13 smithi160 ceph-mon[35859]: pgmap v192: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-09-20T19:43:15.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:15 smithi114 ceph-mon[28260]: pgmap v193: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-09-20T19:43:15.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:15 smithi160 ceph-mon[35859]: pgmap v193: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-09-20T19:43:17.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:17 smithi114 ceph-mon[28260]: pgmap v194: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-09-20T19:43:17.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:17 smithi160 ceph-mon[35859]: pgmap v194: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-09-20T19:43:19.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:19 smithi114 ceph-mon[28260]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-09-20T19:43:19.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:19 smithi160 ceph-mon[35859]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-09-20T19:43:21.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:21 smithi114 ceph-mon[28260]: pgmap v196: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:21.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:21 smithi160 ceph-mon[35859]: pgmap v196: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:23.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:23 smithi114 ceph-mon[28260]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:23.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:23 smithi160 ceph-mon[35859]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:25.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:25 smithi114 ceph-mon[28260]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:25.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:25 smithi160 ceph-mon[35859]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:26.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:26 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:43:26.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:26 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:43:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:27 smithi114 ceph-mon[28260]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:27.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:27 smithi160 ceph-mon[35859]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:29.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:29 smithi114 ceph-mon[28260]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:29.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:29 smithi160 ceph-mon[35859]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:31.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:31 smithi114 ceph-mon[28260]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:31.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:31 smithi160 ceph-mon[35859]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:33.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:33 smithi114 ceph-mon[28260]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:33.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:33 smithi160 ceph-mon[35859]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:35.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:35 smithi114 ceph-mon[28260]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:35.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:35 smithi160 ceph-mon[35859]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:43:37.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:37 smithi114 ceph-mon[28260]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-09-20T19:43:37.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:37 smithi160 ceph-mon[35859]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-09-20T19:43:39.529 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:39 smithi160 ceph-mon[35859]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-09-20T19:43:39.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:39.676+0000 7fe72d7fa640 1 -- 172.21.15.114:0/4265409584 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+89 (secure 0 0 0) 0x7fe7300626e0 con 0x7fe71806e310 2024-09-20T19:43:39.678 INFO:teuthology.orchestra.run.smithi114.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:43:39.678 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:39 smithi114 ceph-mon[28260]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-09-20T19:43:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:39.679+0000 7fe73663e640 1 -- 172.21.15.114:0/4265409584 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fe71806e310 msgr2=0x7fe7180707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:39.679+0000 7fe73663e640 1 --2- 172.21.15.114:0/4265409584 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fe71806e310 0x7fe7180707d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe724005fd0 tx=0x7fe724005e20 comp rx=0 tx=0).stop 2024-09-20T19:43:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:39.679+0000 7fe73663e640 1 -- 172.21.15.114:0/4265409584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe730100880 msgr2=0x7fe7301963e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:39.679+0000 7fe73663e640 1 --2- 172.21.15.114:0/4265409584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe730100880 0x7fe7301963e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe720006010 tx=0x7fe7200048f0 comp rx=0 tx=0).stop 2024-09-20T19:43:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:39.679+0000 7fe73663e640 1 -- 172.21.15.114:0/4265409584 shutdown_connections 2024-09-20T19:43:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:39.679+0000 7fe73663e640 1 --2- 172.21.15.114:0/4265409584 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fe71806e310 0x7fe7180707d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:39.679+0000 7fe73663e640 1 --2- 172.21.15.114:0/4265409584 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe730101a80 0x7fe730196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:39.679+0000 7fe73663e640 1 --2- 172.21.15.114:0/4265409584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe730100880 0x7fe7301963e0 secure :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe720006010 tx=0x7fe7200048f0 comp rx=0 tx=0).stop 2024-09-20T19:43:39.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:39.679+0000 7fe73663e640 1 -- 172.21.15.114:0/4265409584 >> 172.21.15.114:0/4265409584 conn(0x7fe7300fc010 msgr2=0x7fe730104f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:39.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:39.679+0000 7fe73663e640 1 -- 172.21.15.114:0/4265409584 shutdown_connections 2024-09-20T19:43:39.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:39.679+0000 7fe73663e640 1 -- 172.21.15.114:0/4265409584 wait complete. 2024-09-20T19:43:40.423 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-20T19:43:40.676 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:40 smithi114 ceph-mon[28260]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:43:40.676 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:40 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:43:40.677 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:40 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:40.677 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:40 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:40.677 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:40 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:40.677 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:40 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:43:40.677 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:40 smithi114 ceph-mon[28260]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:43:40.677 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:40 smithi114 ceph-mon[28260]: pgmap v206: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-09-20T19:43:40.757 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:43:40.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:40 smithi160 ceph-mon[35859]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:43:40.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:40 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:43:40.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:40 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:40.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:40 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T19:43:40.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:40 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T19:43:40.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:40 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:43:40.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:40 smithi160 ceph-mon[35859]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T19:43:40.780 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:40 smithi160 ceph-mon[35859]: pgmap v206: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-09-20T19:43:41.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.336+0000 7fc234cb0640 1 -- 172.21.15.114:0/32454829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc230101a40 msgr2=0x7fc230101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:41.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.336+0000 7fc234cb0640 1 --2- 172.21.15.114:0/32454829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc230101a40 0x7fc230101ec0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc224009880 tx=0x7fc22402a4a0 comp rx=0 tx=0).stop 2024-09-20T19:43:41.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.337+0000 7fc234cb0640 1 -- 172.21.15.114:0/32454829 shutdown_connections 2024-09-20T19:43:41.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.337+0000 7fc234cb0640 1 --2- 172.21.15.114:0/32454829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc230101a40 0x7fc230101ec0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:41.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.337+0000 7fc234cb0640 1 --2- 172.21.15.114:0/32454829 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc230100840 0x7fc230100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:41.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.337+0000 7fc234cb0640 1 -- 172.21.15.114:0/32454829 >> 172.21.15.114:0/32454829 conn(0x7fc2300fbff0 msgr2=0x7fc2300fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:41.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.337+0000 7fc234cb0640 1 -- 172.21.15.114:0/32454829 shutdown_connections 2024-09-20T19:43:41.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.337+0000 7fc234cb0640 1 -- 172.21.15.114:0/32454829 wait complete. 2024-09-20T19:43:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.338+0000 7fc234cb0640 1 Processor -- start 2024-09-20T19:43:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.338+0000 7fc234cb0640 1 -- start start 2024-09-20T19:43:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.338+0000 7fc234cb0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc230100840 0x7fc23019dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.339+0000 7fc22e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc230100840 0x7fc23019dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.339+0000 7fc22e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc230100840 0x7fc23019dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40448/0 (socket says 172.21.15.114:40448) 2024-09-20T19:43:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.339+0000 7fc22e59c640 1 -- 172.21.15.114:0/2029061091 learned_addr learned my addr 172.21.15.114:0/2029061091 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:43:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.339+0000 7fc234cb0640 1 --2- 172.21.15.114:0/2029061091 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc230101a40 0x7fc23019e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.339+0000 7fc234cb0640 1 -- 172.21.15.114:0/2029061091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc230109680 con 0x7fc230100840 2024-09-20T19:43:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.339+0000 7fc22dd9b640 1 --2- 172.21.15.114:0/2029061091 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc230101a40 0x7fc23019e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.339+0000 7fc234cb0640 1 -- 172.21.15.114:0/2029061091 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc230109500 con 0x7fc230101a40 2024-09-20T19:43:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.340+0000 7fc22e59c640 1 -- 172.21.15.114:0/2029061091 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc230101a40 msgr2=0x7fc23019e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.340+0000 7fc22e59c640 1 --2- 172.21.15.114:0/2029061091 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc230101a40 0x7fc23019e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.340+0000 7fc22e59c640 1 -- 172.21.15.114:0/2029061091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23019e690 con 0x7fc230100840 2024-09-20T19:43:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.340+0000 7fc22e59c640 1 --2- 172.21.15.114:0/2029061091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc230100840 0x7fc23019dad0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc22000eaa0 tx=0x7fc22000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.341+0000 7fc21f7fe640 1 -- 172.21.15.114:0/2029061091 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc22000ca60 con 0x7fc230100840 2024-09-20T19:43:41.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.341+0000 7fc21f7fe640 1 -- 172.21.15.114:0/2029061091 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc22000cc00 con 0x7fc230100840 2024-09-20T19:43:41.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.341+0000 7fc234cb0640 1 -- 172.21.15.114:0/2029061091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2301a30d0 con 0x7fc230100840 2024-09-20T19:43:41.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.341+0000 7fc21f7fe640 1 -- 172.21.15.114:0/2029061091 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc220003e50 con 0x7fc230100840 2024-09-20T19:43:41.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.341+0000 7fc234cb0640 1 -- 172.21.15.114:0/2029061091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2301a3660 con 0x7fc230100840 2024-09-20T19:43:41.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.342+0000 7fc234cb0640 1 -- 172.21.15.114:0/2029061091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc230100c40 con 0x7fc230100840 2024-09-20T19:43:41.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.344+0000 7fc21f7fe640 1 -- 172.21.15.114:0/2029061091 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fc220004000 con 0x7fc230100840 2024-09-20T19:43:41.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.345+0000 7fc21f7fe640 1 --2- 172.21.15.114:0/2029061091 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc21806e5c0 0x7fc218070a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:41.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.345+0000 7fc21f7fe640 1 -- 172.21.15.114:0/2029061091 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7fc22008e830 con 0x7fc230100840 2024-09-20T19:43:41.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.345+0000 7fc22dd9b640 1 --2- 172.21.15.114:0/2029061091 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc21806e5c0 0x7fc218070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:41.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.346+0000 7fc22dd9b640 1 --2- 172.21.15.114:0/2029061091 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc21806e5c0 0x7fc218070a80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc23019ef80 tx=0x7fc224009620 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:41.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.352+0000 7fc21f7fe640 1 -- 172.21.15.114:0/2029061091 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fc22005c1f0 con 0x7fc230100840 2024-09-20T19:43:41.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.668+0000 7fc234cb0640 1 -- 172.21.15.114:0/2029061091 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7fc23006a330 con 0x7fc21806e5c0 2024-09-20T19:43:41.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.669+0000 7fc21f7fe640 1 -- 172.21.15.114:0/2029061091 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7fc23006a330 con 0x7fc21806e5c0 2024-09-20T19:43:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.671+0000 7fc234cb0640 1 -- 172.21.15.114:0/2029061091 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc21806e5c0 msgr2=0x7fc218070a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.671+0000 7fc234cb0640 1 --2- 172.21.15.114:0/2029061091 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc21806e5c0 0x7fc218070a80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc23019ef80 tx=0x7fc224009620 comp rx=0 tx=0).stop 2024-09-20T19:43:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.671+0000 7fc234cb0640 1 -- 172.21.15.114:0/2029061091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc230100840 msgr2=0x7fc23019dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.671+0000 7fc234cb0640 1 --2- 172.21.15.114:0/2029061091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc230100840 0x7fc23019dad0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc22000eaa0 tx=0x7fc22000ef60 comp rx=0 tx=0).stop 2024-09-20T19:43:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.671+0000 7fc234cb0640 1 -- 172.21.15.114:0/2029061091 shutdown_connections 2024-09-20T19:43:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.671+0000 7fc234cb0640 1 --2- 172.21.15.114:0/2029061091 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc21806e5c0 0x7fc218070a80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.671+0000 7fc234cb0640 1 --2- 172.21.15.114:0/2029061091 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc230101a40 0x7fc23019e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.671+0000 7fc234cb0640 1 --2- 172.21.15.114:0/2029061091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc230100840 0x7fc23019dad0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.671+0000 7fc234cb0640 1 -- 172.21.15.114:0/2029061091 >> 172.21.15.114:0/2029061091 conn(0x7fc2300fbff0 msgr2=0x7fc230069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.671+0000 7fc234cb0640 1 -- 172.21.15.114:0/2029061091 shutdown_connections 2024-09-20T19:43:41.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.671+0000 7fc234cb0640 1 -- 172.21.15.114:0/2029061091 wait complete. 2024-09-20T19:43:41.685 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-20T19:43:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.773+0000 7fd8bb5bd640 1 -- 172.21.15.114:0/1059481933 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8b4104d60 msgr2=0x7fd8b4107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.773+0000 7fd8bb5bd640 1 --2- 172.21.15.114:0/1059481933 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8b4104d60 0x7fd8b4107150 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009880 tx=0x7fd8a402a4b0 comp rx=0 tx=0).stop 2024-09-20T19:43:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.773+0000 7fd8bb5bd640 1 -- 172.21.15.114:0/1059481933 shutdown_connections 2024-09-20T19:43:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.773+0000 7fd8bb5bd640 1 --2- 172.21.15.114:0/1059481933 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8b4104d60 0x7fd8b4107150 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.773+0000 7fd8bb5bd640 1 --2- 172.21.15.114:0/1059481933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8b4102430 0x7fd8b4104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.773+0000 7fd8bb5bd640 1 -- 172.21.15.114:0/1059481933 >> 172.21.15.114:0/1059481933 conn(0x7fd8b40fbff0 msgr2=0x7fd8b40fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.773+0000 7fd8bb5bd640 1 -- 172.21.15.114:0/1059481933 shutdown_connections 2024-09-20T19:43:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.774+0000 7fd8bb5bd640 1 -- 172.21.15.114:0/1059481933 wait complete. 2024-09-20T19:43:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.774+0000 7fd8bb5bd640 1 Processor -- start 2024-09-20T19:43:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.775+0000 7fd8bb5bd640 1 -- start start 2024-09-20T19:43:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.775+0000 7fd8bb5bd640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8b4102430 0x7fd8b4197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.775+0000 7fd8bb5bd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8b4104d60 0x7fd8b4197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.775+0000 7fd8bb5bd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b410a700 con 0x7fd8b4104d60 2024-09-20T19:43:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.775+0000 7fd8bb5bd640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b410a580 con 0x7fd8b4102430 2024-09-20T19:43:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.775+0000 7fd8b8b58640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8b4104d60 0x7fd8b4197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.775+0000 7fd8b8b58640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8b4104d60 0x7fd8b4197790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40472/0 (socket says 172.21.15.114:40472) 2024-09-20T19:43:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.775+0000 7fd8b8b58640 1 -- 172.21.15.114:0/2265690498 learned_addr learned my addr 172.21.15.114:0/2265690498 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:43:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.775+0000 7fd8b9359640 1 --2- 172.21.15.114:0/2265690498 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8b4102430 0x7fd8b4197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.776+0000 7fd8b8b58640 1 -- 172.21.15.114:0/2265690498 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8b4102430 msgr2=0x7fd8b4197250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.776+0000 7fd8b8b58640 1 --2- 172.21.15.114:0/2265690498 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8b4102430 0x7fd8b4197250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.776+0000 7fd8b8b58640 1 -- 172.21.15.114:0/2265690498 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b4197e10 con 0x7fd8b4104d60 2024-09-20T19:43:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.776+0000 7fd8b8b58640 1 --2- 172.21.15.114:0/2265690498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8b4104d60 0x7fd8b4197790 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009850 tx=0x7fd8a4032c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.776+0000 7fd8aa7fc640 1 -- 172.21.15.114:0/2265690498 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fd8a403f030 con 0x7fd8b4104d60 2024-09-20T19:43:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.776+0000 7fd8bb5bd640 1 -- 172.21.15.114:0/2265690498 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8b419c850 con 0x7fd8b4104d60 2024-09-20T19:43:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.777+0000 7fd8bb5bd640 1 -- 172.21.15.114:0/2265690498 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8b419cde0 con 0x7fd8b4104d60 2024-09-20T19:43:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.777+0000 7fd8aa7fc640 1 -- 172.21.15.114:0/2265690498 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd8a4004b80 con 0x7fd8b4104d60 2024-09-20T19:43:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.778+0000 7fd8aa7fc640 1 -- 172.21.15.114:0/2265690498 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fd8a4049060 con 0x7fd8b4104d60 2024-09-20T19:43:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.778+0000 7fd8bb5bd640 1 -- 172.21.15.114:0/2265690498 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8b419d0a0 con 0x7fd8b4104d60 2024-09-20T19:43:41.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.780+0000 7fd8aa7fc640 1 -- 172.21.15.114:0/2265690498 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fd8a4042030 con 0x7fd8b4104d60 2024-09-20T19:43:41.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.780+0000 7fd8aa7fc640 1 --2- 172.21.15.114:0/2265690498 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd8a006e310 0x7fd8a00707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:41.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.780+0000 7fd8aa7fc640 1 -- 172.21.15.114:0/2265690498 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7fd8a4040070 con 0x7fd8b4104d60 2024-09-20T19:43:41.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.781+0000 7fd8b9359640 1 --2- 172.21.15.114:0/2265690498 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd8a006e310 0x7fd8a00707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:41.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.782+0000 7fd8b9359640 1 --2- 172.21.15.114:0/2265690498 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd8a006e310 0x7fd8a00707d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00095f0 tx=0x7fd8b0005f90 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:41.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:41.789+0000 7fd8aa7fc640 1 -- 172.21.15.114:0/2265690498 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fd8a407c340 con 0x7fd8b4104d60 2024-09-20T19:43:41.943 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:41 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:43:42.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:41 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:43:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.060+0000 7fd8bb5bd640 1 -- 172.21.15.114:0/2265690498 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7fd8b4105600 con 0x7fd8a006e310 2024-09-20T19:43:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.062+0000 7fd8aa7fc640 1 -- 172.21.15.114:0/2265690498 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7fd8b4105600 con 0x7fd8a006e310 2024-09-20T19:43:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.064+0000 7fd8bb5bd640 1 -- 172.21.15.114:0/2265690498 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd8a006e310 msgr2=0x7fd8a00707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.064+0000 7fd8bb5bd640 1 --2- 172.21.15.114:0/2265690498 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd8a006e310 0x7fd8a00707d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00095f0 tx=0x7fd8b0005f90 comp rx=0 tx=0).stop 2024-09-20T19:43:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.064+0000 7fd8bb5bd640 1 -- 172.21.15.114:0/2265690498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8b4104d60 msgr2=0x7fd8b4197790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.064+0000 7fd8bb5bd640 1 --2- 172.21.15.114:0/2265690498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8b4104d60 0x7fd8b4197790 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009850 tx=0x7fd8a4032c80 comp rx=0 tx=0).stop 2024-09-20T19:43:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.064+0000 7fd8bb5bd640 1 -- 172.21.15.114:0/2265690498 shutdown_connections 2024-09-20T19:43:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.064+0000 7fd8bb5bd640 1 --2- 172.21.15.114:0/2265690498 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fd8a006e310 0x7fd8a00707d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.064+0000 7fd8bb5bd640 1 --2- 172.21.15.114:0/2265690498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8b4104d60 0x7fd8b4197790 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.064+0000 7fd8bb5bd640 1 --2- 172.21.15.114:0/2265690498 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd8b4102430 0x7fd8b4197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.064+0000 7fd8bb5bd640 1 -- 172.21.15.114:0/2265690498 >> 172.21.15.114:0/2265690498 conn(0x7fd8b40fbff0 msgr2=0x7fd8b40fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.065+0000 7fd8bb5bd640 1 -- 172.21.15.114:0/2265690498 shutdown_connections 2024-09-20T19:43:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.065+0000 7fd8bb5bd640 1 -- 172.21.15.114:0/2265690498 wait complete. 2024-09-20T19:43:42.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.150+0000 7f0f2c5ce640 1 -- 172.21.15.114:0/583840230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f24104da0 msgr2=0x7f0f24107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:42.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.150+0000 7f0f2c5ce640 1 --2- 172.21.15.114:0/583840230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f24104da0 0x7f0f24107190 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009880 tx=0x7f0f1402a4a0 comp rx=0 tx=0).stop 2024-09-20T19:43:42.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.151+0000 7f0f2c5ce640 1 -- 172.21.15.114:0/583840230 shutdown_connections 2024-09-20T19:43:42.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.151+0000 7f0f2c5ce640 1 --2- 172.21.15.114:0/583840230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f24104da0 0x7f0f24107190 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:42.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.151+0000 7f0f2c5ce640 1 --2- 172.21.15.114:0/583840230 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0f24102470 0x7f0f24104860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:42.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.151+0000 7f0f2c5ce640 1 -- 172.21.15.114:0/583840230 >> 172.21.15.114:0/583840230 conn(0x7f0f240fc010 msgr2=0x7f0f240fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:42.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.151+0000 7f0f2c5ce640 1 -- 172.21.15.114:0/583840230 shutdown_connections 2024-09-20T19:43:42.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.151+0000 7f0f2c5ce640 1 -- 172.21.15.114:0/583840230 wait complete. 2024-09-20T19:43:42.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.152+0000 7f0f2c5ce640 1 Processor -- start 2024-09-20T19:43:42.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.152+0000 7f0f2c5ce640 1 -- start start 2024-09-20T19:43:42.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.152+0000 7f0f2c5ce640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f24102470 0x7f0f24197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:42.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.152+0000 7f0f2c5ce640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0f24104da0 0x7f0f24197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:42.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.152+0000 7f0f2c5ce640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2410a740 con 0x7f0f24102470 2024-09-20T19:43:42.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.152+0000 7f0f2c5ce640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2410a5c0 con 0x7f0f24104da0 2024-09-20T19:43:42.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.153+0000 7f0f2a36a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f24102470 0x7f0f24197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:42.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.153+0000 7f0f2a36a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f24102470 0x7f0f24197440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40496/0 (socket says 172.21.15.114:40496) 2024-09-20T19:43:42.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.153+0000 7f0f2a36a640 1 -- 172.21.15.114:0/3362831354 learned_addr learned my addr 172.21.15.114:0/3362831354 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:43:42.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.153+0000 7f0f29b69640 1 --2- 172.21.15.114:0/3362831354 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0f24104da0 0x7f0f24197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:42.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.153+0000 7f0f2a36a640 1 -- 172.21.15.114:0/3362831354 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0f24104da0 msgr2=0x7f0f24197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:42.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.153+0000 7f0f2a36a640 1 --2- 172.21.15.114:0/3362831354 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0f24104da0 0x7f0f24197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:42.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.153+0000 7f0f2a36a640 1 -- 172.21.15.114:0/3362831354 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f24198000 con 0x7f0f24102470 2024-09-20T19:43:42.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.154+0000 7f0f2a36a640 1 --2- 172.21.15.114:0/3362831354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f24102470 0x7f0f24197440 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000e970 tx=0x7f0f2000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:42.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.154+0000 7f0f1b7fe640 1 -- 172.21.15.114:0/3362831354 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0f2000cac0 con 0x7f0f24102470 2024-09-20T19:43:42.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.154+0000 7f0f1b7fe640 1 -- 172.21.15.114:0/3362831354 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0f2000cc60 con 0x7f0f24102470 2024-09-20T19:43:42.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.154+0000 7f0f2c5ce640 1 -- 172.21.15.114:0/3362831354 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f241002c0 con 0x7f0f24102470 2024-09-20T19:43:42.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.154+0000 7f0f1b7fe640 1 -- 172.21.15.114:0/3362831354 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0f20018720 con 0x7f0f24102470 2024-09-20T19:43:42.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.154+0000 7f0f2c5ce640 1 -- 172.21.15.114:0/3362831354 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f24100820 con 0x7f0f24102470 2024-09-20T19:43:42.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.155+0000 7f0f2c5ce640 1 -- 172.21.15.114:0/3362831354 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f24102850 con 0x7f0f24102470 2024-09-20T19:43:42.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.161+0000 7f0f1b7fe640 1 -- 172.21.15.114:0/3362831354 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f0f20010430 con 0x7f0f24102470 2024-09-20T19:43:42.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.162+0000 7f0f1b7fe640 1 --2- 172.21.15.114:0/3362831354 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0f1006e420 0x7f0f100708e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:42.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.165+0000 7f0f29b69640 1 --2- 172.21.15.114:0/3362831354 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0f1006e420 0x7f0f100708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:42.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.166+0000 7f0f1b7fe640 1 -- 172.21.15.114:0/3362831354 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f0f2008eb20 con 0x7f0f24102470 2024-09-20T19:43:42.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.166+0000 7f0f29b69640 1 --2- 172.21.15.114:0/3362831354 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0f1006e420 0x7f0f100708e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009230 tx=0x7f0f140058d0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:42.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.166+0000 7f0f1b7fe640 1 -- 172.21.15.114:0/3362831354 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f0f20060190 con 0x7f0f24102470 2024-09-20T19:43:42.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.510+0000 7f0f2c5ce640 1 -- 172.21.15.114:0/3362831354 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f0f24105640 con 0x7f0f1006e420 2024-09-20T19:43:42.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.528+0000 7f0f1b7fe640 1 -- 172.21.15.114:0/3362831354 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3792 (secure 0 0 0) 0x7f0f24105640 con 0x7f0f1006e420 2024-09-20T19:43:42.529 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:43:42.529 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (4m) 43s ago 6m 26.3M - 0.25.0 c8568f914cd2 d19ec799e98c 2024-09-20T19:43:42.530 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (6m) 43s ago 6m 8220k - 19.1.1-369-ged8ee0f2 6acb0a031eab ddd91f706b5f 2024-09-20T19:43:42.530 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi160 smithi160 running (5m) 47s ago 5m 5268k - 19.1.1-369-ged8ee0f2 6acb0a031eab 8ce900c6456f 2024-09-20T19:43:42.530 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (6m) 43s ago 6m 6895k - 19.1.1-369-ged8ee0f2 6acb0a031eab 0e32ab0eeecd 2024-09-20T19:43:42.530 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi160 smithi160 running (5m) 47s ago 5m 6891k - 19.1.1-369-ged8ee0f2 6acb0a031eab d858e40463d6 2024-09-20T19:43:42.530 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (4m) 43s ago 6m 78.8M - 9.4.12 2bacad6d85d8 cd034302ef4b 2024-09-20T19:43:42.530 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.scsmgq smithi114 running (66s) 43s ago 65s 16.3M - 19.1.1-369-ged8ee0f2 6acb0a031eab fcac0f36c17d 2024-09-20T19:43:42.530 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tlrcnf smithi114 running (60s) 43s ago 59s 16.0M - 19.1.1-369-ged8ee0f2 6acb0a031eab db741608c3d0 2024-09-20T19:43:42.530 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi160.lylyja smithi160 running (57s) 47s ago 57s 14.6M - 19.1.1-369-ged8ee0f2 6acb0a031eab aa5d71040f0c 2024-09-20T19:43:42.530 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi160.xirihp smithi160 running (63s) 47s ago 62s 15.4M - 19.1.1-369-ged8ee0f2 6acb0a031eab da5866e71f3a 2024-09-20T19:43:42.530 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.rvhgji smithi114 *:9283,8765,8443 running (7m) 43s ago 7m 498M - 19.1.1-369-ged8ee0f2 6acb0a031eab 59997c1e986a 2024-09-20T19:43:42.530 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi160.bspiak smithi160 *:8443,9283,8765 running (5m) 47s ago 5m 441M - 19.1.1-369-ged8ee0f2 6acb0a031eab 50306543ba66 2024-09-20T19:43:42.530 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (7m) 43s ago 7m 47.9M 2048M 19.1.1-369-ged8ee0f2 6acb0a031eab ed7f013f2513 2024-09-20T19:43:42.531 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi160 smithi160 running (5m) 47s ago 5m 34.6M 2048M 19.1.1-369-ged8ee0f2 6acb0a031eab 961cc10b05cc 2024-09-20T19:43:42.531 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (6m) 43s ago 6m 16.3M - 1.5.0 0da6a335fe13 6542a55ff841 2024-09-20T19:43:42.531 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi160 smithi160 *:9100 running (5m) 47s ago 5m 16.4M - 1.5.0 0da6a335fe13 ff98c99cbe97 2024-09-20T19:43:42.531 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (4m) 43s ago 4m 47.9M 3946M 19.1.1-369-ged8ee0f2 6acb0a031eab 6f0510eaf505 2024-09-20T19:43:42.531 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (3m) 43s ago 3m 53.8M 3946M 19.1.1-369-ged8ee0f2 6acb0a031eab 98b6e5128717 2024-09-20T19:43:42.531 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (3m) 43s ago 3m 46.5M 3946M 19.1.1-369-ged8ee0f2 6acb0a031eab dda945a7e7ca 2024-09-20T19:43:42.531 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi160 running (2m) 47s ago 2m 52.7M 4970M 19.1.1-369-ged8ee0f2 6acb0a031eab 9facdfb1a166 2024-09-20T19:43:42.531 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi160 running (2m) 47s ago 2m 45.3M 4970M 19.1.1-369-ged8ee0f2 6acb0a031eab e15086d3d575 2024-09-20T19:43:42.531 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi160 running (2m) 47s ago 2m 51.3M 4970M 19.1.1-369-ged8ee0f2 6acb0a031eab 7f90f136e99a 2024-09-20T19:43:42.531 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (4m) 43s ago 5m 60.2M - 2.43.0 a07b618ecd1d 733ef913ac0b 2024-09-20T19:43:42.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.531+0000 7f0f2c5ce640 1 -- 172.21.15.114:0/3362831354 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0f1006e420 msgr2=0x7f0f100708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:42.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.531+0000 7f0f2c5ce640 1 --2- 172.21.15.114:0/3362831354 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0f1006e420 0x7f0f100708e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009230 tx=0x7f0f140058d0 comp rx=0 tx=0).stop 2024-09-20T19:43:42.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.531+0000 7f0f2c5ce640 1 -- 172.21.15.114:0/3362831354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f24102470 msgr2=0x7f0f24197440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:42.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.531+0000 7f0f2c5ce640 1 --2- 172.21.15.114:0/3362831354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f24102470 0x7f0f24197440 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000e970 tx=0x7f0f2000ee30 comp rx=0 tx=0).stop 2024-09-20T19:43:42.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.531+0000 7f0f2c5ce640 1 -- 172.21.15.114:0/3362831354 shutdown_connections 2024-09-20T19:43:42.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.531+0000 7f0f2c5ce640 1 --2- 172.21.15.114:0/3362831354 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f0f1006e420 0x7f0f100708e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:42.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.531+0000 7f0f2c5ce640 1 --2- 172.21.15.114:0/3362831354 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0f24104da0 0x7f0f24197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:42.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.531+0000 7f0f2c5ce640 1 --2- 172.21.15.114:0/3362831354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f24102470 0x7f0f24197440 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:42.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.531+0000 7f0f2c5ce640 1 -- 172.21.15.114:0/3362831354 >> 172.21.15.114:0/3362831354 conn(0x7f0f240fc010 msgr2=0x7f0f240fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:42.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.531+0000 7f0f2c5ce640 1 -- 172.21.15.114:0/3362831354 shutdown_connections 2024-09-20T19:43:42.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.531+0000 7f0f2c5ce640 1 -- 172.21.15.114:0/3362831354 wait complete. 2024-09-20T19:43:42.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.614+0000 7f6b5bcf8640 1 -- 172.21.15.114:0/1555295780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b54100b60 msgr2=0x7f6b54100fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:42.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.614+0000 7f6b5bcf8640 1 --2- 172.21.15.114:0/1555295780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b54100b60 0x7f6b54100fc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009880 tx=0x7f6b4402a4a0 comp rx=0 tx=0).stop 2024-09-20T19:43:42.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.614+0000 7f6b5bcf8640 1 -- 172.21.15.114:0/1555295780 shutdown_connections 2024-09-20T19:43:42.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.614+0000 7f6b5bcf8640 1 --2- 172.21.15.114:0/1555295780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b54100b60 0x7f6b54100fc0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:42.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.614+0000 7f6b5bcf8640 1 --2- 172.21.15.114:0/1555295780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b54100220 0x7f6b54100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:42.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.614+0000 7f6b5bcf8640 1 -- 172.21.15.114:0/1555295780 >> 172.21.15.114:0/1555295780 conn(0x7f6b540fbff0 msgr2=0x7f6b540fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:42.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.614+0000 7f6b5bcf8640 1 -- 172.21.15.114:0/1555295780 shutdown_connections 2024-09-20T19:43:42.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.614+0000 7f6b5bcf8640 1 -- 172.21.15.114:0/1555295780 wait complete. 2024-09-20T19:43:42.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.615+0000 7f6b5bcf8640 1 Processor -- start 2024-09-20T19:43:42.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.615+0000 7f6b5bcf8640 1 -- start start 2024-09-20T19:43:42.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.615+0000 7f6b5bcf8640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b54100220 0x7f6b54105d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:42.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.615+0000 7f6b5bcf8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b54100b60 0x7f6b54106270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:42.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.615+0000 7f6b5bcf8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5410c970 con 0x7f6b54100b60 2024-09-20T19:43:42.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.615+0000 7f6b5bcf8640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5410c7f0 con 0x7f6b54100220 2024-09-20T19:43:42.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.616+0000 7f6b59293640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b54100b60 0x7f6b54106270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:42.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.616+0000 7f6b59293640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b54100b60 0x7f6b54106270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40512/0 (socket says 172.21.15.114:40512) 2024-09-20T19:43:42.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.616+0000 7f6b59293640 1 -- 172.21.15.114:0/857752857 learned_addr learned my addr 172.21.15.114:0/857752857 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:43:42.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.616+0000 7f6b59a94640 1 --2- 172.21.15.114:0/857752857 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b54100220 0x7f6b54105d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:42.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.616+0000 7f6b59293640 1 -- 172.21.15.114:0/857752857 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b54100220 msgr2=0x7f6b54105d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:42.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.616+0000 7f6b59293640 1 --2- 172.21.15.114:0/857752857 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b54100220 0x7f6b54105d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:42.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.616+0000 7f6b59293640 1 -- 172.21.15.114:0/857752857 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b541028b0 con 0x7f6b54100b60 2024-09-20T19:43:42.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.616+0000 7f6b59293640 1 --2- 172.21.15.114:0/857752857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b54100b60 0x7f6b54106270 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6b44006010 tx=0x7f6b44004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:42.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.616+0000 7f6b4affd640 1 -- 172.21.15.114:0/857752857 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6b4403f070 con 0x7f6b54100b60 2024-09-20T19:43:42.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.617+0000 7f6b4affd640 1 -- 172.21.15.114:0/857752857 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6b440049d0 con 0x7f6b54100b60 2024-09-20T19:43:42.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.617+0000 7f6b5bcf8640 1 -- 172.21.15.114:0/857752857 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b54102b40 con 0x7f6b54100b60 2024-09-20T19:43:42.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.617+0000 7f6b5bcf8640 1 -- 172.21.15.114:0/857752857 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b54103070 con 0x7f6b54100b60 2024-09-20T19:43:42.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.617+0000 7f6b4affd640 1 -- 172.21.15.114:0/857752857 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6b44035030 con 0x7f6b54100b60 2024-09-20T19:43:42.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.618+0000 7f6b5bcf8640 1 -- 172.21.15.114:0/857752857 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b54100620 con 0x7f6b54100b60 2024-09-20T19:43:42.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.619+0000 7f6b4affd640 1 -- 172.21.15.114:0/857752857 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f6b44042070 con 0x7f6b54100b60 2024-09-20T19:43:42.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.619+0000 7f6b4affd640 1 --2- 172.21.15.114:0/857752857 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f6b4006e2b0 0x7f6b40070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:42.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.620+0000 7f6b4affd640 1 -- 172.21.15.114:0/857752857 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f6b440aeda0 con 0x7f6b54100b60 2024-09-20T19:43:42.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.620+0000 7f6b59a94640 1 --2- 172.21.15.114:0/857752857 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f6b4006e2b0 0x7f6b40070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:42.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.620+0000 7f6b59a94640 1 --2- 172.21.15.114:0/857752857 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f6b4006e2b0 0x7f6b40070770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6b50009c20 tx=0x7f6b50005f90 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:42.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:42.624+0000 7f6b4affd640 1 -- 172.21.15.114:0/857752857 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f6b44080310 con 0x7f6b54100b60 2024-09-20T19:43:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:42 smithi114 ceph-mon[28260]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:42 smithi114 ceph-mon[28260]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:42 smithi114 ceph-mon[28260]: pgmap v207: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-09-20T19:43:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:42 smithi114 ceph-mon[28260]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:42 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:43:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:42 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:43.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:42 smithi160 ceph-mon[35859]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:43.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:42 smithi160 ceph-mon[35859]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:43.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:42 smithi160 ceph-mon[35859]: pgmap v207: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-09-20T19:43:43.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:42 smithi160 ceph-mon[35859]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:43.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:42 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' 2024-09-20T19:43:43.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:42 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T19:43:43.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.066+0000 7f6b5bcf8640 1 -- 172.21.15.114:0/857752857 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) -- 0x7f6b5402d8c0 con 0x7f6b54100b60 2024-09-20T19:43:43.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.067+0000 7f6b4affd640 1 -- 172.21.15.114:0/857752857 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) ==== 56+0+633 (secure 0 0 0) 0x7f6b44084b40 con 0x7f6b54100b60 2024-09-20T19:43:43.069 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-20T19:43:43.069 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-09-20T19:43:43.069 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)": 2 2024-09-20T19:43:43.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-20T19:43:43.069 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-09-20T19:43:43.069 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)": 2 2024-09-20T19:43:43.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-20T19:43:43.069 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-09-20T19:43:43.069 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)": 6 2024-09-20T19:43:43.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-20T19:43:43.070 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-09-20T19:43:43.070 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)": 4 2024-09-20T19:43:43.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-20T19:43:43.070 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-09-20T19:43:43.070 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)": 14 2024-09-20T19:43:43.070 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-20T19:43:43.070 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-20T19:43:43.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.070+0000 7f6b5bcf8640 1 -- 172.21.15.114:0/857752857 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f6b4006e2b0 msgr2=0x7f6b40070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.070+0000 7f6b5bcf8640 1 --2- 172.21.15.114:0/857752857 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f6b4006e2b0 0x7f6b40070770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6b50009c20 tx=0x7f6b50005f90 comp rx=0 tx=0).stop 2024-09-20T19:43:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.071+0000 7f6b5bcf8640 1 -- 172.21.15.114:0/857752857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b54100b60 msgr2=0x7f6b54106270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.071+0000 7f6b5bcf8640 1 --2- 172.21.15.114:0/857752857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b54100b60 0x7f6b54106270 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6b44006010 tx=0x7f6b44004570 comp rx=0 tx=0).stop 2024-09-20T19:43:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.071+0000 7f6b5bcf8640 1 -- 172.21.15.114:0/857752857 shutdown_connections 2024-09-20T19:43:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.071+0000 7f6b5bcf8640 1 --2- 172.21.15.114:0/857752857 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f6b4006e2b0 0x7f6b40070770 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.071+0000 7f6b5bcf8640 1 --2- 172.21.15.114:0/857752857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b54100b60 0x7f6b54106270 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.071+0000 7f6b5bcf8640 1 --2- 172.21.15.114:0/857752857 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b54100220 0x7f6b54105d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.071+0000 7f6b5bcf8640 1 -- 172.21.15.114:0/857752857 >> 172.21.15.114:0/857752857 conn(0x7f6b540fbff0 msgr2=0x7f6b54108a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.071+0000 7f6b5bcf8640 1 -- 172.21.15.114:0/857752857 shutdown_connections 2024-09-20T19:43:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.071+0000 7f6b5bcf8640 1 -- 172.21.15.114:0/857752857 wait complete. 2024-09-20T19:43:43.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.200+0000 7f792a6c9640 1 -- 172.21.15.114:0/2891385741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7924100840 msgr2=0x7f7924100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:43.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.200+0000 7f792a6c9640 1 --2- 172.21.15.114:0/2891385741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7924100840 0x7f7924100c40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7914009880 tx=0x7f791402a4a0 comp rx=0 tx=0).stop 2024-09-20T19:43:43.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.200+0000 7f792a6c9640 1 -- 172.21.15.114:0/2891385741 shutdown_connections 2024-09-20T19:43:43.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.200+0000 7f792a6c9640 1 --2- 172.21.15.114:0/2891385741 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7924101a40 0x7f7924101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:43.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.200+0000 7f792a6c9640 1 --2- 172.21.15.114:0/2891385741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7924100840 0x7f7924100c40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:43.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.200+0000 7f792a6c9640 1 -- 172.21.15.114:0/2891385741 >> 172.21.15.114:0/2891385741 conn(0x7f79240fbff0 msgr2=0x7f79240fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:43.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.200+0000 7f792a6c9640 1 -- 172.21.15.114:0/2891385741 shutdown_connections 2024-09-20T19:43:43.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.200+0000 7f792a6c9640 1 -- 172.21.15.114:0/2891385741 wait complete. 2024-09-20T19:43:43.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.201+0000 7f792a6c9640 1 Processor -- start 2024-09-20T19:43:43.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.201+0000 7f792a6c9640 1 -- start start 2024-09-20T19:43:43.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.201+0000 7f792a6c9640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7924100840 0x7f7924104760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:43.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.202+0000 7f792a6c9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7924101a40 0x7f7924102e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:43.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.202+0000 7f792a6c9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7924109680 con 0x7f7924101a40 2024-09-20T19:43:43.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.202+0000 7f792a6c9640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f7924109500 con 0x7f7924100840 2024-09-20T19:43:43.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.202+0000 7f79237fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7924101a40 0x7f7924102e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:43.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.202+0000 7f7923fff640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7924100840 0x7f7924104760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:43.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.202+0000 7f7923fff640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7924100840 0x7f7924104760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.114:55286/0 (socket says 172.21.15.114:55286) 2024-09-20T19:43:43.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.202+0000 7f79237fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7924101a40 0x7f7924102e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40536/0 (socket says 172.21.15.114:40536) 2024-09-20T19:43:43.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.202+0000 7f7923fff640 1 -- 172.21.15.114:0/2473072088 learned_addr learned my addr 172.21.15.114:0/2473072088 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:43:43.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.202+0000 7f7923fff640 1 -- 172.21.15.114:0/2473072088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7924101a40 msgr2=0x7f7924102e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:43.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.202+0000 7f7923fff640 1 --2- 172.21.15.114:0/2473072088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7924101a40 0x7f7924102e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:43.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.202+0000 7f7923fff640 1 -- 172.21.15.114:0/2473072088 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792419b110 con 0x7f7924100840 2024-09-20T19:43:43.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.202+0000 7f7923fff640 1 --2- 172.21.15.114:0/2473072088 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7924100840 0x7f7924104760 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7914006010 tx=0x7f79140045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:43.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.202+0000 7f79217fa640 1 -- 172.21.15.114:0/2473072088 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f791403f070 con 0x7f7924100840 2024-09-20T19:43:43.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.203+0000 7f79217fa640 1 -- 172.21.15.114:0/2473072088 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f79140049d0 con 0x7f7924100840 2024-09-20T19:43:43.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.203+0000 7f792a6c9640 1 -- 172.21.15.114:0/2473072088 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7924103360 con 0x7f7924100840 2024-09-20T19:43:43.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.203+0000 7f79217fa640 1 -- 172.21.15.114:0/2473072088 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f7914035030 con 0x7f7924100840 2024-09-20T19:43:43.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.203+0000 7f792a6c9640 1 -- 172.21.15.114:0/2473072088 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7924103890 con 0x7f7924100840 2024-09-20T19:43:43.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.204+0000 7f792a6c9640 1 -- 172.21.15.114:0/2473072088 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7924109e60 con 0x7f7924100840 2024-09-20T19:43:43.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.205+0000 7f79217fa640 1 -- 172.21.15.114:0/2473072088 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f7914042070 con 0x7f7924100840 2024-09-20T19:43:43.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.205+0000 7f79217fa640 1 --2- 172.21.15.114:0/2473072088 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f790c06e310 0x7f790c0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:43:43.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.206+0000 7f79217fa640 1 -- 172.21.15.114:0/2473072088 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f79140aecb0 con 0x7f7924100840 2024-09-20T19:43:43.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.206+0000 7f79237fe640 1 --2- 172.21.15.114:0/2473072088 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f790c06e310 0x7f790c0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:43:43.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.206+0000 7f79237fe640 1 --2- 172.21.15.114:0/2473072088 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f790c06e310 0x7f790c0707d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f79241044e0 tx=0x7f7918005f90 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:43:43.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.214+0000 7f79217fa640 1 -- 172.21.15.114:0/2473072088 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f7914080220 con 0x7f7924100840 2024-09-20T19:43:43.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.446+0000 7f792a6c9640 1 -- 172.21.15.114:0/2473072088 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7f792406a270 con 0x7f790c06e310 2024-09-20T19:43:43.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.448+0000 7f79217fa640 1 -- 172.21.15.114:0/2473072088 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+439 (secure 0 0 0) 0x7f792406a270 con 0x7f790c06e310 2024-09-20T19:43:43.450 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-20T19:43:43.450 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:43:43.450 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-09-20T19:43:43.450 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-20T19:43:43.451 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [], 2024-09-20T19:43:43.451 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "0/2 daemons upgraded", 2024-09-20T19:43:43.451 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d image on host smithi160", 2024-09-20T19:43:43.451 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-09-20T19:43:43.451 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-20T19:43:43.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.451+0000 7f792a6c9640 1 -- 172.21.15.114:0/2473072088 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f790c06e310 msgr2=0x7f790c0707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:43.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.451+0000 7f792a6c9640 1 --2- 172.21.15.114:0/2473072088 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f790c06e310 0x7f790c0707d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f79241044e0 tx=0x7f7918005f90 comp rx=0 tx=0).stop 2024-09-20T19:43:43.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.451+0000 7f792a6c9640 1 -- 172.21.15.114:0/2473072088 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7924100840 msgr2=0x7f7924104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:43:43.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.451+0000 7f792a6c9640 1 --2- 172.21.15.114:0/2473072088 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7924100840 0x7f7924104760 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7914006010 tx=0x7f79140045e0 comp rx=0 tx=0).stop 2024-09-20T19:43:43.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.451+0000 7f792a6c9640 1 -- 172.21.15.114:0/2473072088 shutdown_connections 2024-09-20T19:43:43.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.452+0000 7f792a6c9640 1 --2- 172.21.15.114:0/2473072088 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f790c06e310 0x7f790c0707d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:43.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.452+0000 7f792a6c9640 1 --2- 172.21.15.114:0/2473072088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7924101a40 0x7f7924102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:43.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.452+0000 7f792a6c9640 1 --2- 172.21.15.114:0/2473072088 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7924100840 0x7f7924104760 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:43:43.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.452+0000 7f792a6c9640 1 -- 172.21.15.114:0/2473072088 >> 172.21.15.114:0/2473072088 conn(0x7f79240fbff0 msgr2=0x7f7924069b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:43:43.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.452+0000 7f792a6c9640 1 -- 172.21.15.114:0/2473072088 shutdown_connections 2024-09-20T19:43:43.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:43:43.452+0000 7f792a6c9640 1 -- 172.21.15.114:0/2473072088 wait complete. 2024-09-20T19:43:43.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:43 smithi114 ceph-mon[28260]: Upgrade: Target is version 19.3.0-5127-ge8baca1e (squid) 2024-09-20T19:43:43.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:43 smithi114 ceph-mon[28260]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d'] 2024-09-20T19:43:43.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:43 smithi114 ceph-mon[28260]: Upgrade: Need to upgrade myself (mgr.smithi114.rvhgji) 2024-09-20T19:43:43.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:43 smithi114 ceph-mon[28260]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d on smithi160 2024-09-20T19:43:43.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:43 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/857752857' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:43:43.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:43 smithi114 ceph-mon[28260]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:44.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:43 smithi160 ceph-mon[35859]: Upgrade: Target is version 19.3.0-5127-ge8baca1e (squid) 2024-09-20T19:43:44.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:43 smithi160 ceph-mon[35859]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d'] 2024-09-20T19:43:44.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:43 smithi160 ceph-mon[35859]: Upgrade: Need to upgrade myself (mgr.smithi114.rvhgji) 2024-09-20T19:43:44.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:43 smithi160 ceph-mon[35859]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d on smithi160 2024-09-20T19:43:44.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:43 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/857752857' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:43:44.030 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:43 smithi160 ceph-mon[35859]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:43:44.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:44 smithi114 ceph-mon[28260]: pgmap v208: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-20T19:43:45.029 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:44 smithi160 ceph-mon[35859]: pgmap v208: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-20T19:43:47.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:47 smithi114 ceph-mon[28260]: pgmap v209: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-20T19:43:47.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:47 smithi160 ceph-mon[35859]: pgmap v209: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-20T19:43:49.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:49 smithi114 ceph-mon[28260]: pgmap v210: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-09-20T19:43:49.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:49 smithi160 ceph-mon[35859]: pgmap v210: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-09-20T19:43:51.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:51 smithi114 ceph-mon[28260]: pgmap v211: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-09-20T19:43:51.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:51 smithi160 ceph-mon[35859]: pgmap v211: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-09-20T19:43:53.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:53 smithi114 ceph-mon[28260]: pgmap v212: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-09-20T19:43:53.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:53 smithi160 ceph-mon[35859]: pgmap v212: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-09-20T19:43:55.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:55 smithi114 ceph-mon[28260]: pgmap v213: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-09-20T19:43:55.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:55 smithi160 ceph-mon[35859]: pgmap v213: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-09-20T19:43:56.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:56 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:43:56.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:56 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:43:57.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:57 smithi114 ceph-mon[28260]: pgmap v214: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-09-20T19:43:57.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:57 smithi160 ceph-mon[35859]: pgmap v214: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-09-20T19:43:59.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:43:59 smithi114 ceph-mon[28260]: pgmap v215: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-09-20T19:43:59.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:43:59 smithi160 ceph-mon[35859]: pgmap v215: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-09-20T19:44:01.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:01 smithi114 ceph-mon[28260]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:44:01.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:01 smithi160 ceph-mon[35859]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:44:03.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:03 smithi114 ceph-mon[28260]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:44:03.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:03 smithi160 ceph-mon[35859]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:44:05.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:05 smithi114 ceph-mon[28260]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:44:05.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:05 smithi160 ceph-mon[35859]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:44:06.028 INFO:tasks.workunit.client.1.smithi160.stderr:Updating files: 95% (12162/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-09-20T19:44:06.156 INFO:tasks.workunit.client.0.smithi114.stderr:Note: switching to 'e8baca1eac6c3c9c88bf2ea4493e448bd54936a7'. 2024-09-20T19:44:06.156 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-20T19:44:06.156 INFO:tasks.workunit.client.0.smithi114.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-20T19:44:06.157 INFO:tasks.workunit.client.0.smithi114.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-20T19:44:06.157 INFO:tasks.workunit.client.0.smithi114.stderr:state without impacting any branches by switching back to a branch. 2024-09-20T19:44:06.157 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-20T19:44:06.157 INFO:tasks.workunit.client.0.smithi114.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-20T19:44:06.157 INFO:tasks.workunit.client.0.smithi114.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-20T19:44:06.157 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-20T19:44:06.157 INFO:tasks.workunit.client.0.smithi114.stderr: git switch -c 2024-09-20T19:44:06.157 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-20T19:44:06.157 INFO:tasks.workunit.client.0.smithi114.stderr:Or undo this operation with: 2024-09-20T19:44:06.157 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-20T19:44:06.157 INFO:tasks.workunit.client.0.smithi114.stderr: git switch - 2024-09-20T19:44:06.157 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-20T19:44:06.157 INFO:tasks.workunit.client.0.smithi114.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-20T19:44:06.158 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-20T19:44:06.158 INFO:tasks.workunit.client.0.smithi114.stderr:HEAD is now at e8baca1eac6 Merge branch 'alertmanager-custom-webhook' of https://github.com/adk3798/ceph into wip-adk-testing-2024-09-20-0910 2024-09-20T19:44:06.162 DEBUG:teuthology.orchestra.run.smithi114:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-20T19:44:06.238 INFO:tasks.workunit.client.0.smithi114.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-20T19:44:06.240 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-20T19:44:06.240 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-20T19:44:06.372 INFO:tasks.workunit.client.1.smithi160.stderr:Note: switching to 'e8baca1eac6c3c9c88bf2ea4493e448bd54936a7'. 2024-09-20T19:44:06.372 INFO:tasks.workunit.client.1.smithi160.stderr: 2024-09-20T19:44:06.372 INFO:tasks.workunit.client.1.smithi160.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-20T19:44:06.372 INFO:tasks.workunit.client.1.smithi160.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-20T19:44:06.372 INFO:tasks.workunit.client.1.smithi160.stderr:state without impacting any branches by switching back to a branch. 2024-09-20T19:44:06.372 INFO:tasks.workunit.client.1.smithi160.stderr: 2024-09-20T19:44:06.372 INFO:tasks.workunit.client.1.smithi160.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-20T19:44:06.373 INFO:tasks.workunit.client.1.smithi160.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-20T19:44:06.373 INFO:tasks.workunit.client.1.smithi160.stderr: 2024-09-20T19:44:06.373 INFO:tasks.workunit.client.1.smithi160.stderr: git switch -c 2024-09-20T19:44:06.373 INFO:tasks.workunit.client.1.smithi160.stderr: 2024-09-20T19:44:06.373 INFO:tasks.workunit.client.1.smithi160.stderr:Or undo this operation with: 2024-09-20T19:44:06.373 INFO:tasks.workunit.client.1.smithi160.stderr: 2024-09-20T19:44:06.373 INFO:tasks.workunit.client.1.smithi160.stderr: git switch - 2024-09-20T19:44:06.373 INFO:tasks.workunit.client.1.smithi160.stderr: 2024-09-20T19:44:06.373 INFO:tasks.workunit.client.1.smithi160.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-20T19:44:06.373 INFO:tasks.workunit.client.1.smithi160.stderr: 2024-09-20T19:44:06.373 INFO:tasks.workunit.client.1.smithi160.stderr:HEAD is now at e8baca1eac6 Merge branch 'alertmanager-custom-webhook' of https://github.com/adk3798/ceph into wip-adk-testing-2024-09-20-0910 2024-09-20T19:44:06.377 DEBUG:teuthology.orchestra.run.smithi160:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-09-20T19:44:06.438 INFO:tasks.workunit.client.1.smithi160.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-20T19:44:06.440 INFO:tasks.workunit.client.1.smithi160.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-09-20T19:44:06.440 INFO:tasks.workunit.client.1.smithi160.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-20T19:44:07.142 INFO:tasks.workunit.client.1.smithi160.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-20T19:44:07.204 INFO:tasks.workunit.client.1.smithi160.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-20T19:44:07.244 INFO:tasks.workunit.client.1.smithi160.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-09-20T19:44:07.246 INFO:tasks.workunit.client.1.smithi160.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-09-20T19:44:07.246 INFO:tasks.workunit.client.1.smithi160.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-20T19:44:07.285 INFO:tasks.workunit.client.1.smithi160.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-09-20T19:44:07.288 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-20T19:44:07.289 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-09-20T19:44:07.344 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-09-20T19:44:07.346 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-09-20T19:44:07.346 DEBUG:teuthology.orchestra.run.smithi160:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-09-20T19:44:07.406 INFO:tasks.workunit.client.1.smithi160.stderr:+ mkdir -p fsstress 2024-09-20T19:44:07.408 INFO:tasks.workunit.client.1.smithi160.stderr:+ pushd fsstress 2024-09-20T19:44:07.408 INFO:tasks.workunit.client.1.smithi160.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-09-20T19:44:07.409 INFO:tasks.workunit.client.1.smithi160.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-09-20T19:44:07.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:07 smithi114 ceph-mon[28260]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:44:07.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:07 smithi160 ceph-mon[35859]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:44:08.543 INFO:tasks.workunit.client.1.smithi160.stderr:+ tar xzf ltp-full.tgz 2024-09-20T19:44:09.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:09 smithi114 ceph-mon[28260]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:44:09.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:09 smithi160 ceph-mon[35859]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-09-20T19:44:11.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:11 smithi114 ceph-mon[28260]: pgmap v221: 65 pgs: 65 active+clean; 1.6 MiB data, 176 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s wr, 0 op/s 2024-09-20T19:44:11.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:11 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:44:11.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:11 smithi160 ceph-mon[35859]: pgmap v221: 65 pgs: 65 active+clean; 1.6 MiB data, 176 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s wr, 0 op/s 2024-09-20T19:44:11.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:11 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:44:13.253 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-20T19:44:13.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.542+0000 7f878c531640 1 -- 172.21.15.114:0/390442817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8784102450 msgr2=0x7f8784104840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:13.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.542+0000 7f878c531640 1 --2- 172.21.15.114:0/390442817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8784102450 0x7f8784104840 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8780009880 tx=0x7f878002a4a0 comp rx=0 tx=0).stop 2024-09-20T19:44:13.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.542+0000 7f878c531640 1 -- 172.21.15.114:0/390442817 shutdown_connections 2024-09-20T19:44:13.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.542+0000 7f878c531640 1 --2- 172.21.15.114:0/390442817 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8784104d80 0x7f8784107170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:13.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.542+0000 7f878c531640 1 --2- 172.21.15.114:0/390442817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8784102450 0x7f8784104840 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:13.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.542+0000 7f878c531640 1 -- 172.21.15.114:0/390442817 >> 172.21.15.114:0/390442817 conn(0x7f87840fbff0 msgr2=0x7f87840fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:13.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.542+0000 7f878c531640 1 -- 172.21.15.114:0/390442817 shutdown_connections 2024-09-20T19:44:13.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.542+0000 7f878c531640 1 -- 172.21.15.114:0/390442817 wait complete. 2024-09-20T19:44:13.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.543+0000 7f878c531640 1 Processor -- start 2024-09-20T19:44:13.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.543+0000 7f878c531640 1 -- start start 2024-09-20T19:44:13.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.543+0000 7f878c531640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8784102450 0x7f8784197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:13.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.543+0000 7f878c531640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8784104d80 0x7f8784197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:13.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.543+0000 7f878c531640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f878410a7b0 con 0x7f8784104d80 2024-09-20T19:44:13.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.543+0000 7f878c531640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f878410a630 con 0x7f8784102450 2024-09-20T19:44:13.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.543+0000 7f878a2cd640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8784102450 0x7f8784197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:13.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.543+0000 7f878a2cd640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8784102450 0x7f8784197500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.114:35134/0 (socket says 172.21.15.114:35134) 2024-09-20T19:44:13.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.543+0000 7f878a2cd640 1 -- 172.21.15.114:0/3779560036 learned_addr learned my addr 172.21.15.114:0/3779560036 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:44:13.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.543+0000 7f8789acc640 1 --2- 172.21.15.114:0/3779560036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8784104d80 0x7f8784197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:13.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.544+0000 7f878a2cd640 1 -- 172.21.15.114:0/3779560036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8784104d80 msgr2=0x7f8784197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:13.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.544+0000 7f878a2cd640 1 --2- 172.21.15.114:0/3779560036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8784104d80 0x7f8784197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:13.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.544+0000 7f878a2cd640 1 -- 172.21.15.114:0/3779560036 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87841980c0 con 0x7f8784102450 2024-09-20T19:44:13.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.544+0000 7f878a2cd640 1 --2- 172.21.15.114:0/3779560036 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8784102450 0x7f8784197500 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8780006010 tx=0x7f87800048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:13.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.544+0000 7f877b7fe640 1 -- 172.21.15.114:0/3779560036 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f878003f070 con 0x7f8784102450 2024-09-20T19:44:13.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.544+0000 7f878c531640 1 -- 172.21.15.114:0/3779560036 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f878419caa0 con 0x7f8784102450 2024-09-20T19:44:13.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.544+0000 7f877b7fe640 1 -- 172.21.15.114:0/3779560036 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f87800044d0 con 0x7f8784102450 2024-09-20T19:44:13.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.544+0000 7f877b7fe640 1 -- 172.21.15.114:0/3779560036 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8780035030 con 0x7f8784102450 2024-09-20T19:44:13.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.544+0000 7f878c531640 1 -- 172.21.15.114:0/3779560036 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f878419d030 con 0x7f8784102450 2024-09-20T19:44:13.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.545+0000 7f878c531640 1 -- 172.21.15.114:0/3779560036 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8784102830 con 0x7f8784102450 2024-09-20T19:44:13.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.546+0000 7f877b7fe640 1 -- 172.21.15.114:0/3779560036 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f8780042070 con 0x7f8784102450 2024-09-20T19:44:13.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.547+0000 7f877b7fe640 1 --2- 172.21.15.114:0/3779560036 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f877006e310 0x7f87700707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:13.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.547+0000 7f8789acc640 1 --2- 172.21.15.114:0/3779560036 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f877006e310 0x7f87700707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:13.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.547+0000 7f877b7fe640 1 -- 172.21.15.114:0/3779560036 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f87800aede0 con 0x7f8784102450 2024-09-20T19:44:13.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.547+0000 7f8789acc640 1 --2- 172.21.15.114:0/3779560036 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f877006e310 0x7f87700707d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f87841989b0 tx=0x7f877400a250 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:13.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.553+0000 7f877b7fe640 1 -- 172.21.15.114:0/3779560036 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f87800803d0 con 0x7f8784102450 2024-09-20T19:44:13.569 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-20T19:44:13.714 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-20T19:44:13.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:13 smithi114 ceph-mon[28260]: pgmap v222: 65 pgs: 65 active+clean; 1.6 MiB data, 180 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s wr, 0 op/s 2024-09-20T19:44:13.716 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-20T19:44:13.716 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-20T19:44:13.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:13 smithi160 ceph-mon[35859]: pgmap v222: 65 pgs: 65 active+clean; 1.6 MiB data, 180 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s wr, 0 op/s 2024-09-20T19:44:13.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.824+0000 7f878c531640 1 -- 172.21.15.114:0/3779560036 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7f8784105700 con 0x7f877006e310 2024-09-20T19:44:13.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.825+0000 7f877b7fe640 1 -- 172.21.15.114:0/3779560036 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+439 (secure 0 0 0) 0x7f8784105700 con 0x7f877006e310 2024-09-20T19:44:13.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.828+0000 7f878c531640 1 -- 172.21.15.114:0/3779560036 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f877006e310 msgr2=0x7f87700707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:13.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.828+0000 7f878c531640 1 --2- 172.21.15.114:0/3779560036 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f877006e310 0x7f87700707d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f87841989b0 tx=0x7f877400a250 comp rx=0 tx=0).stop 2024-09-20T19:44:13.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.828+0000 7f878c531640 1 -- 172.21.15.114:0/3779560036 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8784102450 msgr2=0x7f8784197500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:13.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.828+0000 7f878c531640 1 --2- 172.21.15.114:0/3779560036 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8784102450 0x7f8784197500 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8780006010 tx=0x7f87800048f0 comp rx=0 tx=0).stop 2024-09-20T19:44:13.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.828+0000 7f878c531640 1 -- 172.21.15.114:0/3779560036 shutdown_connections 2024-09-20T19:44:13.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.828+0000 7f878c531640 1 --2- 172.21.15.114:0/3779560036 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f877006e310 0x7f87700707d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:13.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.828+0000 7f878c531640 1 --2- 172.21.15.114:0/3779560036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8784104d80 0x7f8784197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:13.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.828+0000 7f878c531640 1 --2- 172.21.15.114:0/3779560036 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8784102450 0x7f8784197500 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:13.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.828+0000 7f878c531640 1 -- 172.21.15.114:0/3779560036 >> 172.21.15.114:0/3779560036 conn(0x7f87840fbff0 msgr2=0x7f87840fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:13.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.829+0000 7f878c531640 1 -- 172.21.15.114:0/3779560036 shutdown_connections 2024-09-20T19:44:13.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.829+0000 7f878c531640 1 -- 172.21.15.114:0/3779560036 wait complete. 2024-09-20T19:44:13.839 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-20T19:44:13.871 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-20T19:44:13.875 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T19:44:13.875 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-20T19:44:13.892 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-09-20T19:44:13.893 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-09-20T19:44:13.893 DEBUG:teuthology.orchestra.run.smithi114:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-09-20T19:44:13.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.918+0000 7f2b3d8da640 1 -- 172.21.15.114:0/224535732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b38104d00 msgr2=0x7f2b381070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:13.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.918+0000 7f2b3d8da640 1 --2- 172.21.15.114:0/224535732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b38104d00 0x7f2b381070f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c0098b0 tx=0x7f2b2c02a4d0 comp rx=0 tx=0).stop 2024-09-20T19:44:13.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.918+0000 7f2b3d8da640 1 -- 172.21.15.114:0/224535732 shutdown_connections 2024-09-20T19:44:13.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.918+0000 7f2b3d8da640 1 --2- 172.21.15.114:0/224535732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b38104d00 0x7f2b381070f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:13.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.918+0000 7f2b3d8da640 1 --2- 172.21.15.114:0/224535732 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2b381023d0 0x7f2b381047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:13.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.918+0000 7f2b3d8da640 1 -- 172.21.15.114:0/224535732 >> 172.21.15.114:0/224535732 conn(0x7f2b380fbff0 msgr2=0x7f2b380fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:13.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.918+0000 7f2b3d8da640 1 -- 172.21.15.114:0/224535732 shutdown_connections 2024-09-20T19:44:13.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.918+0000 7f2b3d8da640 1 -- 172.21.15.114:0/224535732 wait complete. 2024-09-20T19:44:13.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.918+0000 7f2b3d8da640 1 Processor -- start 2024-09-20T19:44:13.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.919+0000 7f2b3d8da640 1 -- start start 2024-09-20T19:44:13.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.919+0000 7f2b3d8da640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b381023d0 0x7f2b38197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:13.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.919+0000 7f2b3d8da640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2b38104d00 0x7f2b38197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:13.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.919+0000 7f2b3d8da640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3810a790 con 0x7f2b381023d0 2024-09-20T19:44:13.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.919+0000 7f2b3d8da640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3810a610 con 0x7f2b38104d00 2024-09-20T19:44:13.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.919+0000 7f2b36ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b381023d0 0x7f2b38197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:13.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.919+0000 7f2b36ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b381023d0 0x7f2b38197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41638/0 (socket says 172.21.15.114:41638) 2024-09-20T19:44:13.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.919+0000 7f2b36ffd640 1 -- 172.21.15.114:0/1215699973 learned_addr learned my addr 172.21.15.114:0/1215699973 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:44:13.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.919+0000 7f2b367fc640 1 --2- 172.21.15.114:0/1215699973 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2b38104d00 0x7f2b38197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:13.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.919+0000 7f2b36ffd640 1 -- 172.21.15.114:0/1215699973 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2b38104d00 msgr2=0x7f2b38197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:13.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.919+0000 7f2b36ffd640 1 --2- 172.21.15.114:0/1215699973 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2b38104d00 0x7f2b38197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:13.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.919+0000 7f2b36ffd640 1 -- 172.21.15.114:0/1215699973 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b381980e0 con 0x7f2b381023d0 2024-09-20T19:44:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.920+0000 7f2b36ffd640 1 --2- 172.21.15.114:0/1215699973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b381023d0 0x7f2b38197520 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800d6f0 tx=0x7f2b2800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.920+0000 7f2b3c8d8640 1 -- 172.21.15.114:0/1215699973 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2b28017070 con 0x7f2b381023d0 2024-09-20T19:44:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.920+0000 7f2b3d8da640 1 -- 172.21.15.114:0/1215699973 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b3819cb20 con 0x7f2b381023d0 2024-09-20T19:44:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.920+0000 7f2b3c8d8640 1 -- 172.21.15.114:0/1215699973 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2b2800de00 con 0x7f2b381023d0 2024-09-20T19:44:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.920+0000 7f2b3c8d8640 1 -- 172.21.15.114:0/1215699973 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2b28004bb0 con 0x7f2b381023d0 2024-09-20T19:44:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.920+0000 7f2b3d8da640 1 -- 172.21.15.114:0/1215699973 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b3819d080 con 0x7f2b381023d0 2024-09-20T19:44:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.921+0000 7f2b3d8da640 1 -- 172.21.15.114:0/1215699973 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b381027b0 con 0x7f2b381023d0 2024-09-20T19:44:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.922+0000 7f2b3c8d8640 1 -- 172.21.15.114:0/1215699973 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f2b28010460 con 0x7f2b381023d0 2024-09-20T19:44:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.922+0000 7f2b3c8d8640 1 --2- 172.21.15.114:0/1215699973 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2b2006e2b0 0x7f2b20070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.922+0000 7f2b3c8d8640 1 -- 172.21.15.114:0/1215699973 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f2b2808dd20 con 0x7f2b381023d0 2024-09-20T19:44:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.923+0000 7f2b367fc640 1 --2- 172.21.15.114:0/1215699973 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2b2006e2b0 0x7f2b20070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.923+0000 7f2b367fc640 1 --2- 172.21.15.114:0/1215699973 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2b2006e2b0 0x7f2b20070770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c009880 tx=0x7f2b2c0058d0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:13.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:13.927+0000 7f2b3c8d8640 1 -- 172.21.15.114:0/1215699973 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f2b2805f330 con 0x7f2b381023d0 2024-09-20T19:44:13.946 INFO:tasks.workunit.client.0.smithi114.stderr:+ mkdir -p fsstress 2024-09-20T19:44:13.949 INFO:tasks.workunit.client.0.smithi114.stderr:+ pushd fsstress 2024-09-20T19:44:13.949 INFO:tasks.workunit.client.0.smithi114.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-09-20T19:44:13.950 INFO:tasks.workunit.client.0.smithi114.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-09-20T19:44:14.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.161+0000 7f2b3d8da640 1 -- 172.21.15.114:0/1215699973 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7f2b381056e0 con 0x7f2b2006e2b0 2024-09-20T19:44:14.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.163+0000 7f2b3c8d8640 1 -- 172.21.15.114:0/1215699973 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+439 (secure 0 0 0) 0x7f2b381056e0 con 0x7f2b2006e2b0 2024-09-20T19:44:14.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.165+0000 7f2b3d8da640 1 -- 172.21.15.114:0/1215699973 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2b2006e2b0 msgr2=0x7f2b20070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:14.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.165+0000 7f2b3d8da640 1 --2- 172.21.15.114:0/1215699973 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2b2006e2b0 0x7f2b20070770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c009880 tx=0x7f2b2c0058d0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.165+0000 7f2b3d8da640 1 -- 172.21.15.114:0/1215699973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b381023d0 msgr2=0x7f2b38197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.165+0000 7f2b3d8da640 1 --2- 172.21.15.114:0/1215699973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b381023d0 0x7f2b38197520 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800d6f0 tx=0x7f2b2800dbb0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.165+0000 7f2b3d8da640 1 -- 172.21.15.114:0/1215699973 shutdown_connections 2024-09-20T19:44:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.165+0000 7f2b3d8da640 1 --2- 172.21.15.114:0/1215699973 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f2b2006e2b0 0x7f2b20070770 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.165+0000 7f2b3d8da640 1 --2- 172.21.15.114:0/1215699973 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2b38104d00 0x7f2b38197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.165+0000 7f2b3d8da640 1 --2- 172.21.15.114:0/1215699973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b381023d0 0x7f2b38197520 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.165+0000 7f2b3d8da640 1 -- 172.21.15.114:0/1215699973 >> 172.21.15.114:0/1215699973 conn(0x7f2b380fbff0 msgr2=0x7f2b380fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.165+0000 7f2b3d8da640 1 -- 172.21.15.114:0/1215699973 shutdown_connections 2024-09-20T19:44:14.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.165+0000 7f2b3d8da640 1 -- 172.21.15.114:0/1215699973 wait complete. 2024-09-20T19:44:14.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.253+0000 7fabf23b6640 1 -- 172.21.15.114:0/1183270295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabec100840 msgr2=0x7fabec100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:14.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.253+0000 7fabf23b6640 1 --2- 172.21.15.114:0/1183270295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabec100840 0x7fabec100c40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009880 tx=0x7fabdc02a4a0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.253+0000 7fabf23b6640 1 -- 172.21.15.114:0/1183270295 shutdown_connections 2024-09-20T19:44:14.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.253+0000 7fabf23b6640 1 --2- 172.21.15.114:0/1183270295 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fabec101a40 0x7fabec101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.253+0000 7fabf23b6640 1 --2- 172.21.15.114:0/1183270295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabec100840 0x7fabec100c40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.253+0000 7fabf23b6640 1 -- 172.21.15.114:0/1183270295 >> 172.21.15.114:0/1183270295 conn(0x7fabec0fbff0 msgr2=0x7fabec0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:14.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.254+0000 7fabf23b6640 1 -- 172.21.15.114:0/1183270295 shutdown_connections 2024-09-20T19:44:14.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.254+0000 7fabf23b6640 1 -- 172.21.15.114:0/1183270295 wait complete. 2024-09-20T19:44:14.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.254+0000 7fabf23b6640 1 Processor -- start 2024-09-20T19:44:14.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.254+0000 7fabf23b6640 1 -- start start 2024-09-20T19:44:14.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.255+0000 7fabf23b6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabec100840 0x7fabec06a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:14.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.255+0000 7fabf23b6640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fabec101a40 0x7fabec19c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:14.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.255+0000 7fabf23b6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fabec109660 con 0x7fabec100840 2024-09-20T19:44:14.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.255+0000 7fabf23b6640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fabec1094e0 con 0x7fabec101a40 2024-09-20T19:44:14.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.255+0000 7fabebfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabec100840 0x7fabec06a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:14.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.255+0000 7fabebfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabec100840 0x7fabec06a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41658/0 (socket says 172.21.15.114:41658) 2024-09-20T19:44:14.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.255+0000 7fabebfff640 1 -- 172.21.15.114:0/1448301255 learned_addr learned my addr 172.21.15.114:0/1448301255 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:44:14.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.255+0000 7fabeb7fe640 1 --2- 172.21.15.114:0/1448301255 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fabec101a40 0x7fabec19c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:14.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.255+0000 7fabeb7fe640 1 -- 172.21.15.114:0/1448301255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabec100840 msgr2=0x7fabec06a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:14.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.255+0000 7fabeb7fe640 1 --2- 172.21.15.114:0/1448301255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabec100840 0x7fabec06a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.255+0000 7fabeb7fe640 1 -- 172.21.15.114:0/1448301255 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabec19c7b0 con 0x7fabec101a40 2024-09-20T19:44:14.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.255+0000 7fabebfff640 1 --2- 172.21.15.114:0/1448301255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabec100840 0x7fabec06a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-20T19:44:14.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.255+0000 7fabeb7fe640 1 --2- 172.21.15.114:0/1448301255 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fabec101a40 0x7fabec19c1f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fabe000e9a0 tx=0x7fabe000ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:14.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.256+0000 7fabe97fa640 1 -- 172.21.15.114:0/1448301255 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fabe000cac0 con 0x7fabec101a40 2024-09-20T19:44:14.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.256+0000 7fabf23b6640 1 -- 172.21.15.114:0/1448301255 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabec19caa0 con 0x7fabec101a40 2024-09-20T19:44:14.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.256+0000 7fabe97fa640 1 -- 172.21.15.114:0/1448301255 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fabe000cc60 con 0x7fabec101a40 2024-09-20T19:44:14.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.256+0000 7fabf23b6640 1 -- 172.21.15.114:0/1448301255 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabec19d030 con 0x7fabec101a40 2024-09-20T19:44:14.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.256+0000 7fabe97fa640 1 -- 172.21.15.114:0/1448301255 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fabe00186c0 con 0x7fabec101a40 2024-09-20T19:44:14.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.257+0000 7fabf23b6640 1 -- 172.21.15.114:0/1448301255 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabd80052c0 con 0x7fabec101a40 2024-09-20T19:44:14.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.257+0000 7fabe97fa640 1 -- 172.21.15.114:0/1448301255 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fabe00189c0 con 0x7fabec101a40 2024-09-20T19:44:14.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.258+0000 7fabe97fa640 1 --2- 172.21.15.114:0/1448301255 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fabd406e2b0 0x7fabd4070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:14.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.258+0000 7fabe97fa640 1 -- 172.21.15.114:0/1448301255 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7fabe008e520 con 0x7fabec101a40 2024-09-20T19:44:14.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.258+0000 7fabebfff640 1 --2- 172.21.15.114:0/1448301255 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fabd406e2b0 0x7fabd4070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:14.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.258+0000 7fabebfff640 1 --2- 172.21.15.114:0/1448301255 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fabd406e2b0 0x7fabd4070770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fabdc0096c0 tx=0x7fabdc0058d0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:14.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.262+0000 7fabe97fa640 1 -- 172.21.15.114:0/1448301255 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fabe005faa0 con 0x7fabec101a40 2024-09-20T19:44:14.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.468+0000 7fabf23b6640 1 -- 172.21.15.114:0/1448301255 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fabd8000bf0 con 0x7fabd406e2b0 2024-09-20T19:44:14.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.478+0000 7fabe97fa640 1 -- 172.21.15.114:0/1448301255 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3792 (secure 0 0 0) 0x7fabd8000bf0 con 0x7fabd406e2b0 2024-09-20T19:44:14.480 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:44:14.480 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (5m) 75s ago 6m 26.3M - 0.25.0 c8568f914cd2 d19ec799e98c 2024-09-20T19:44:14.480 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (7m) 75s ago 7m 8220k - 19.1.1-369-ged8ee0f2 6acb0a031eab ddd91f706b5f 2024-09-20T19:44:14.480 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi160 smithi160 running (6m) 79s ago 6m 5268k - 19.1.1-369-ged8ee0f2 6acb0a031eab 8ce900c6456f 2024-09-20T19:44:14.480 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (6m) 75s ago 6m 6895k - 19.1.1-369-ged8ee0f2 6acb0a031eab 0e32ab0eeecd 2024-09-20T19:44:14.480 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi160 smithi160 running (6m) 79s ago 6m 6891k - 19.1.1-369-ged8ee0f2 6acb0a031eab d858e40463d6 2024-09-20T19:44:14.480 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (5m) 75s ago 6m 78.8M - 9.4.12 2bacad6d85d8 cd034302ef4b 2024-09-20T19:44:14.481 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.scsmgq smithi114 running (98s) 75s ago 97s 16.3M - 19.1.1-369-ged8ee0f2 6acb0a031eab fcac0f36c17d 2024-09-20T19:44:14.481 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tlrcnf smithi114 running (92s) 75s ago 91s 16.0M - 19.1.1-369-ged8ee0f2 6acb0a031eab db741608c3d0 2024-09-20T19:44:14.481 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi160.lylyja smithi160 running (89s) 79s ago 89s 14.6M - 19.1.1-369-ged8ee0f2 6acb0a031eab aa5d71040f0c 2024-09-20T19:44:14.481 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi160.xirihp smithi160 running (95s) 79s ago 94s 15.4M - 19.1.1-369-ged8ee0f2 6acb0a031eab da5866e71f3a 2024-09-20T19:44:14.481 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.rvhgji smithi114 *:9283,8765,8443 running (8m) 75s ago 8m 498M - 19.1.1-369-ged8ee0f2 6acb0a031eab 59997c1e986a 2024-09-20T19:44:14.481 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi160.bspiak smithi160 *:8443,9283,8765 running (5m) 79s ago 5m 441M - 19.1.1-369-ged8ee0f2 6acb0a031eab 50306543ba66 2024-09-20T19:44:14.481 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (8m) 75s ago 8m 47.9M 2048M 19.1.1-369-ged8ee0f2 6acb0a031eab ed7f013f2513 2024-09-20T19:44:14.481 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi160 smithi160 running (5m) 79s ago 5m 34.6M 2048M 19.1.1-369-ged8ee0f2 6acb0a031eab 961cc10b05cc 2024-09-20T19:44:14.481 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (6m) 75s ago 6m 16.3M - 1.5.0 0da6a335fe13 6542a55ff841 2024-09-20T19:44:14.481 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi160 smithi160 *:9100 running (6m) 79s ago 6m 16.4M - 1.5.0 0da6a335fe13 ff98c99cbe97 2024-09-20T19:44:14.481 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (4m) 75s ago 4m 47.9M 3946M 19.1.1-369-ged8ee0f2 6acb0a031eab 6f0510eaf505 2024-09-20T19:44:14.481 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (4m) 75s ago 4m 53.8M 3946M 19.1.1-369-ged8ee0f2 6acb0a031eab 98b6e5128717 2024-09-20T19:44:14.481 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (3m) 75s ago 3m 46.5M 3946M 19.1.1-369-ged8ee0f2 6acb0a031eab dda945a7e7ca 2024-09-20T19:44:14.481 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi160 running (3m) 79s ago 3m 52.7M 4970M 19.1.1-369-ged8ee0f2 6acb0a031eab 9facdfb1a166 2024-09-20T19:44:14.482 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi160 running (3m) 79s ago 3m 45.3M 4970M 19.1.1-369-ged8ee0f2 6acb0a031eab e15086d3d575 2024-09-20T19:44:14.482 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi160 running (2m) 79s ago 2m 51.3M 4970M 19.1.1-369-ged8ee0f2 6acb0a031eab 7f90f136e99a 2024-09-20T19:44:14.482 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (4m) 75s ago 6m 60.2M - 2.43.0 a07b618ecd1d 733ef913ac0b 2024-09-20T19:44:14.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.481+0000 7fabf23b6640 1 -- 172.21.15.114:0/1448301255 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fabd406e2b0 msgr2=0x7fabd4070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:14.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.481+0000 7fabf23b6640 1 --2- 172.21.15.114:0/1448301255 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fabd406e2b0 0x7fabd4070770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fabdc0096c0 tx=0x7fabdc0058d0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.481+0000 7fabf23b6640 1 -- 172.21.15.114:0/1448301255 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fabec101a40 msgr2=0x7fabec19c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:14.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.481+0000 7fabf23b6640 1 --2- 172.21.15.114:0/1448301255 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fabec101a40 0x7fabec19c1f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fabe000e9a0 tx=0x7fabe000ee60 comp rx=0 tx=0).stop 2024-09-20T19:44:14.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.481+0000 7fabf23b6640 1 -- 172.21.15.114:0/1448301255 shutdown_connections 2024-09-20T19:44:14.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.481+0000 7fabf23b6640 1 --2- 172.21.15.114:0/1448301255 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fabd406e2b0 0x7fabd4070770 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.481+0000 7fabf23b6640 1 --2- 172.21.15.114:0/1448301255 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fabec101a40 0x7fabec19c1f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.481+0000 7fabf23b6640 1 --2- 172.21.15.114:0/1448301255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabec100840 0x7fabec06a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.481+0000 7fabf23b6640 1 -- 172.21.15.114:0/1448301255 >> 172.21.15.114:0/1448301255 conn(0x7fabec0fbff0 msgr2=0x7fabec104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:14.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.481+0000 7fabf23b6640 1 -- 172.21.15.114:0/1448301255 shutdown_connections 2024-09-20T19:44:14.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.481+0000 7fabf23b6640 1 -- 172.21.15.114:0/1448301255 wait complete. 2024-09-20T19:44:14.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.566+0000 7f061ddd6640 1 -- 172.21.15.114:0/3697061648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0618104d00 msgr2=0x7f06181070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:14.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.566+0000 7f061ddd6640 1 --2- 172.21.15.114:0/3697061648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0618104d00 0x7f06181070f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f060c0098b0 tx=0x7f060c02a4d0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.566+0000 7f061ddd6640 1 -- 172.21.15.114:0/3697061648 shutdown_connections 2024-09-20T19:44:14.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.566+0000 7f061ddd6640 1 --2- 172.21.15.114:0/3697061648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0618104d00 0x7f06181070f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.566+0000 7f061ddd6640 1 --2- 172.21.15.114:0/3697061648 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f06181023d0 0x7f06181047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.566+0000 7f061ddd6640 1 -- 172.21.15.114:0/3697061648 >> 172.21.15.114:0/3697061648 conn(0x7f06180fbff0 msgr2=0x7f06180fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:14.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.566+0000 7f061ddd6640 1 -- 172.21.15.114:0/3697061648 shutdown_connections 2024-09-20T19:44:14.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.566+0000 7f061ddd6640 1 -- 172.21.15.114:0/3697061648 wait complete. 2024-09-20T19:44:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.567+0000 7f061ddd6640 1 Processor -- start 2024-09-20T19:44:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.567+0000 7f061ddd6640 1 -- start start 2024-09-20T19:44:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.567+0000 7f061ddd6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06181023d0 0x7f0618197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.567+0000 7f061ddd6640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0618104d00 0x7f0618197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.567+0000 7f061ddd6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f061810a790 con 0x7f06181023d0 2024-09-20T19:44:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.567+0000 7f061ddd6640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f061810a610 con 0x7f0618104d00 2024-09-20T19:44:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.567+0000 7f06177fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06181023d0 0x7f0618197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.567+0000 7f06177fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06181023d0 0x7f0618197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41676/0 (socket says 172.21.15.114:41676) 2024-09-20T19:44:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.567+0000 7f06177fe640 1 -- 172.21.15.114:0/286616079 learned_addr learned my addr 172.21.15.114:0/286616079 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:44:14.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.567+0000 7f0616ffd640 1 --2- 172.21.15.114:0/286616079 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0618104d00 0x7f0618197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:14.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.568+0000 7f06177fe640 1 -- 172.21.15.114:0/286616079 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0618104d00 msgr2=0x7f0618197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:14.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.568+0000 7f06177fe640 1 --2- 172.21.15.114:0/286616079 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0618104d00 0x7f0618197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.568+0000 7f06177fe640 1 -- 172.21.15.114:0/286616079 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06181980e0 con 0x7f06181023d0 2024-09-20T19:44:14.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.568+0000 7f0616ffd640 1 --2- 172.21.15.114:0/286616079 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0618104d00 0x7f0618197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-20T19:44:14.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.568+0000 7f06177fe640 1 --2- 172.21.15.114:0/286616079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06181023d0 0x7f0618197520 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f060800e970 tx=0x7f060800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:14.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.568+0000 7f0614ff9640 1 -- 172.21.15.114:0/286616079 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f060800cac0 con 0x7f06181023d0 2024-09-20T19:44:14.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.568+0000 7f0614ff9640 1 -- 172.21.15.114:0/286616079 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f06080054e0 con 0x7f06181023d0 2024-09-20T19:44:14.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.568+0000 7f061ddd6640 1 -- 172.21.15.114:0/286616079 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f061819cb20 con 0x7f06181023d0 2024-09-20T19:44:14.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.568+0000 7f0614ff9640 1 -- 172.21.15.114:0/286616079 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0608010430 con 0x7f06181023d0 2024-09-20T19:44:14.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.568+0000 7f061ddd6640 1 -- 172.21.15.114:0/286616079 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f061819d080 con 0x7f06181023d0 2024-09-20T19:44:14.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.569+0000 7f061ddd6640 1 -- 172.21.15.114:0/286616079 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f061810af70 con 0x7f06181023d0 2024-09-20T19:44:14.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.570+0000 7f0614ff9640 1 -- 172.21.15.114:0/286616079 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f0608004f00 con 0x7f06181023d0 2024-09-20T19:44:14.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.571+0000 7f0614ff9640 1 --2- 172.21.15.114:0/286616079 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f060006e2b0 0x7f0600070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:14.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.571+0000 7f0614ff9640 1 -- 172.21.15.114:0/286616079 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f060808e800 con 0x7f06181023d0 2024-09-20T19:44:14.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.571+0000 7f0616ffd640 1 --2- 172.21.15.114:0/286616079 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f060006e2b0 0x7f0600070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:14.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.571+0000 7f0616ffd640 1 --2- 172.21.15.114:0/286616079 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f060006e2b0 0x7f0600070770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f060c009880 tx=0x7f060c0058d0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:14.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.575+0000 7f0614ff9640 1 -- 172.21.15.114:0/286616079 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f060805c160 con 0x7f06181023d0 2024-09-20T19:44:14.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.838+0000 7f061ddd6640 1 -- 172.21.15.114:0/286616079 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) -- 0x7f061819d910 con 0x7f06181023d0 2024-09-20T19:44:14.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.839+0000 7f0614ff9640 1 -- 172.21.15.114:0/286616079 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) ==== 56+0+633 (secure 0 0 0) 0x7f060805fe10 con 0x7f06181023d0 2024-09-20T19:44:14.840 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-20T19:44:14.840 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-09-20T19:44:14.841 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)": 2 2024-09-20T19:44:14.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-20T19:44:14.841 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-09-20T19:44:14.841 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)": 2 2024-09-20T19:44:14.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-20T19:44:14.841 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-09-20T19:44:14.841 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)": 6 2024-09-20T19:44:14.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-20T19:44:14.841 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-09-20T19:44:14.841 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)": 4 2024-09-20T19:44:14.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-20T19:44:14.841 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-09-20T19:44:14.841 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)": 14 2024-09-20T19:44:14.841 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-20T19:44:14.842 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-20T19:44:14.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.841+0000 7f061ddd6640 1 -- 172.21.15.114:0/286616079 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f060006e2b0 msgr2=0x7f0600070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:14.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.841+0000 7f061ddd6640 1 --2- 172.21.15.114:0/286616079 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f060006e2b0 0x7f0600070770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f060c009880 tx=0x7f060c0058d0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.841+0000 7f061ddd6640 1 -- 172.21.15.114:0/286616079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06181023d0 msgr2=0x7f0618197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:14.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.841+0000 7f061ddd6640 1 --2- 172.21.15.114:0/286616079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06181023d0 0x7f0618197520 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f060800e970 tx=0x7f060800ed70 comp rx=0 tx=0).stop 2024-09-20T19:44:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.841+0000 7f061ddd6640 1 -- 172.21.15.114:0/286616079 shutdown_connections 2024-09-20T19:44:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.841+0000 7f061ddd6640 1 --2- 172.21.15.114:0/286616079 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f060006e2b0 0x7f0600070770 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.841+0000 7f061ddd6640 1 --2- 172.21.15.114:0/286616079 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0618104d00 0x7f0618197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.841+0000 7f061ddd6640 1 --2- 172.21.15.114:0/286616079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06181023d0 0x7f0618197520 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.841+0000 7f061ddd6640 1 -- 172.21.15.114:0/286616079 >> 172.21.15.114:0/286616079 conn(0x7f06180fbff0 msgr2=0x7f06180fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.841+0000 7f061ddd6640 1 -- 172.21.15.114:0/286616079 shutdown_connections 2024-09-20T19:44:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.841+0000 7f061ddd6640 1 -- 172.21.15.114:0/286616079 wait complete. 2024-09-20T19:44:14.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.931+0000 7f38be144640 1 -- 172.21.15.114:0/3443259117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38b8074b40 msgr2=0x7f38b8074fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:14.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.931+0000 7f38be144640 1 --2- 172.21.15.114:0/3443259117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38b8074b40 0x7f38b8074fa0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009880 tx=0x7f38ac02a4a0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.931+0000 7f38be144640 1 -- 172.21.15.114:0/3443259117 shutdown_connections 2024-09-20T19:44:14.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.931+0000 7f38be144640 1 --2- 172.21.15.114:0/3443259117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38b8074b40 0x7f38b8074fa0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.931+0000 7f38be144640 1 --2- 172.21.15.114:0/3443259117 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f38b80761a0 0x7f38b8074600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.931+0000 7f38be144640 1 -- 172.21.15.114:0/3443259117 >> 172.21.15.114:0/3443259117 conn(0x7f38b80fd140 msgr2=0x7f38b80ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:14.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.932+0000 7f38be144640 1 -- 172.21.15.114:0/3443259117 shutdown_connections 2024-09-20T19:44:14.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.932+0000 7f38be144640 1 -- 172.21.15.114:0/3443259117 wait complete. 2024-09-20T19:44:14.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.932+0000 7f38be144640 1 Processor -- start 2024-09-20T19:44:14.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.932+0000 7f38be144640 1 -- start start 2024-09-20T19:44:14.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.932+0000 7f38be144640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f38b8074b40 0x7f38b8199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:14.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.932+0000 7f38be144640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38b80761a0 0x7f38b8199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:14.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.932+0000 7f38be144640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f38b810c9a0 con 0x7f38b80761a0 2024-09-20T19:44:14.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.932+0000 7f38be144640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f38b810c820 con 0x7f38b8074b40 2024-09-20T19:44:14.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.933+0000 7f38b6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38b80761a0 0x7f38b8199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:14.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.933+0000 7f38b77fe640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f38b8074b40 0x7f38b8199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:14.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.933+0000 7f38b6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38b80761a0 0x7f38b8199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41692/0 (socket says 172.21.15.114:41692) 2024-09-20T19:44:14.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.933+0000 7f38b6ffd640 1 -- 172.21.15.114:0/2652044917 learned_addr learned my addr 172.21.15.114:0/2652044917 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:44:14.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.933+0000 7f38b77fe640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f38b8074b40 0x7f38b8199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.114:35198/0 (socket says 172.21.15.114:35198) 2024-09-20T19:44:14.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.933+0000 7f38b77fe640 1 -- 172.21.15.114:0/2652044917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38b80761a0 msgr2=0x7f38b8199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:14.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.933+0000 7f38b77fe640 1 --2- 172.21.15.114:0/2652044917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38b80761a0 0x7f38b8199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:14.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.933+0000 7f38b77fe640 1 -- 172.21.15.114:0/2652044917 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b819a240 con 0x7f38b8074b40 2024-09-20T19:44:14.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.933+0000 7f38b77fe640 1 --2- 172.21.15.114:0/2652044917 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f38b8074b40 0x7f38b8199680 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f38a800d6a0 tx=0x7f38a800db60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:14.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.933+0000 7f38b4ff9640 1 -- 172.21.15.114:0/2652044917 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f38a8017070 con 0x7f38b8074b40 2024-09-20T19:44:14.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.933+0000 7f38b4ff9640 1 -- 172.21.15.114:0/2652044917 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f38a80054e0 con 0x7f38b8074b40 2024-09-20T19:44:14.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.933+0000 7f38be144640 1 -- 172.21.15.114:0/2652044917 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b819ec80 con 0x7f38b8074b40 2024-09-20T19:44:14.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.933+0000 7f38b4ff9640 1 -- 172.21.15.114:0/2652044917 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f38a8004b90 con 0x7f38b8074b40 2024-09-20T19:44:14.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.934+0000 7f38be144640 1 -- 172.21.15.114:0/2652044917 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b819f1c0 con 0x7f38b8074b40 2024-09-20T19:44:14.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.935+0000 7f38be144640 1 -- 172.21.15.114:0/2652044917 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38b8074fa0 con 0x7f38b8074b40 2024-09-20T19:44:14.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.936+0000 7f38b4ff9640 1 -- 172.21.15.114:0/2652044917 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f38a8004e40 con 0x7f38b8074b40 2024-09-20T19:44:14.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.936+0000 7f38b4ff9640 1 --2- 172.21.15.114:0/2652044917 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f38a006e310 0x7f38a00707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:14.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.936+0000 7f38b4ff9640 1 -- 172.21.15.114:0/2652044917 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f38a808dc50 con 0x7f38b8074b40 2024-09-20T19:44:14.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.936+0000 7f38b6ffd640 1 --2- 172.21.15.114:0/2652044917 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f38a006e310 0x7f38a00707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:14.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.937+0000 7f38b6ffd640 1 --2- 172.21.15.114:0/2652044917 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f38a006e310 0x7f38a00707d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f38ac0096c0 tx=0x7f38ac005a20 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:14.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:14.941+0000 7f38b4ff9640 1 -- 172.21.15.114:0/2652044917 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f38a805f280 con 0x7f38b8074b40 2024-09-20T19:44:15.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:15.180+0000 7f38be144640 1 -- 172.21.15.114:0/2652044917 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7f38b80626f0 con 0x7f38a006e310 2024-09-20T19:44:15.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:15.182+0000 7f38b4ff9640 1 -- 172.21.15.114:0/2652044917 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+439 (secure 0 0 0) 0x7f38b80626f0 con 0x7f38a006e310 2024-09-20T19:44:15.184 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-20T19:44:15.184 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:44:15.184 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-09-20T19:44:15.184 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-20T19:44:15.184 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [], 2024-09-20T19:44:15.184 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "0/2 daemons upgraded", 2024-09-20T19:44:15.184 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d image on host smithi160", 2024-09-20T19:44:15.184 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-09-20T19:44:15.184 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-20T19:44:15.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:15.184+0000 7f38be144640 1 -- 172.21.15.114:0/2652044917 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f38a006e310 msgr2=0x7f38a00707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:15.184+0000 7f38be144640 1 --2- 172.21.15.114:0/2652044917 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f38a006e310 0x7f38a00707d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f38ac0096c0 tx=0x7f38ac005a20 comp rx=0 tx=0).stop 2024-09-20T19:44:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:15.184+0000 7f38be144640 1 -- 172.21.15.114:0/2652044917 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f38b8074b40 msgr2=0x7f38b8199680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:15.184+0000 7f38be144640 1 --2- 172.21.15.114:0/2652044917 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f38b8074b40 0x7f38b8199680 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f38a800d6a0 tx=0x7f38a800db60 comp rx=0 tx=0).stop 2024-09-20T19:44:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:15.184+0000 7f38be144640 1 -- 172.21.15.114:0/2652044917 shutdown_connections 2024-09-20T19:44:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:15.184+0000 7f38be144640 1 --2- 172.21.15.114:0/2652044917 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f38a006e310 0x7f38a00707d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:15.184+0000 7f38be144640 1 --2- 172.21.15.114:0/2652044917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38b80761a0 0x7f38b8199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:15.184+0000 7f38be144640 1 --2- 172.21.15.114:0/2652044917 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f38b8074b40 0x7f38b8199680 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:15.184+0000 7f38be144640 1 -- 172.21.15.114:0/2652044917 >> 172.21.15.114:0/2652044917 conn(0x7f38b80fd140 msgr2=0x7f38b8107a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:15.185+0000 7f38be144640 1 -- 172.21.15.114:0/2652044917 shutdown_connections 2024-09-20T19:44:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:15.185+0000 7f38be144640 1 -- 172.21.15.114:0/2652044917 wait complete. 2024-09-20T19:44:15.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:15 smithi114 ceph-mon[28260]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:15.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:15 smithi114 ceph-mon[28260]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:15.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:15 smithi114 ceph-mon[28260]: pgmap v223: 65 pgs: 65 active+clean; 13 MiB data, 211 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s wr, 70 op/s 2024-09-20T19:44:15.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:15 smithi114 ceph-mon[28260]: from='client.24357 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:15.467 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:15 smithi114 ceph-mon[28260]: from='client.? 172.21.15.114:0/286616079' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:44:15.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:15 smithi160 ceph-mon[35859]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:15.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:15 smithi160 ceph-mon[35859]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:15.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:15 smithi160 ceph-mon[35859]: pgmap v223: 65 pgs: 65 active+clean; 13 MiB data, 211 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s wr, 70 op/s 2024-09-20T19:44:15.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:15 smithi160 ceph-mon[35859]: from='client.24357 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:15.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:15 smithi160 ceph-mon[35859]: from='client.? 172.21.15.114:0/286616079' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T19:44:16.091 INFO:tasks.workunit.client.0.smithi114.stderr:+ tar xzf ltp-full.tgz 2024-09-20T19:44:16.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:16 smithi114 ceph-mon[28260]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:16.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:16 smithi160 ceph-mon[35859]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T19:44:17.718 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:17 smithi114 ceph-mon[28260]: pgmap v224: 65 pgs: 65 active+clean; 26 MiB data, 257 MiB used, 536 GiB / 536 GiB avail; 2.2 MiB/s wr, 178 op/s 2024-09-20T19:44:17.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:17 smithi160 ceph-mon[35859]: pgmap v224: 65 pgs: 65 active+clean; 26 MiB data, 257 MiB used, 536 GiB / 536 GiB avail; 2.2 MiB/s wr, 178 op/s 2024-09-20T19:44:19.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:19 smithi114 ceph-mon[28260]: pgmap v225: 65 pgs: 65 active+clean; 40 MiB data, 329 MiB used, 536 GiB / 536 GiB avail; 3.5 MiB/s wr, 289 op/s 2024-09-20T19:44:19.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:19 smithi160 ceph-mon[35859]: pgmap v225: 65 pgs: 65 active+clean; 40 MiB data, 329 MiB used, 536 GiB / 536 GiB avail; 3.5 MiB/s wr, 289 op/s 2024-09-20T19:44:21.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:21 smithi114 ceph-mon[28260]: pgmap v226: 65 pgs: 65 active+clean; 56 MiB data, 349 MiB used, 536 GiB / 536 GiB avail; 4.8 MiB/s wr, 330 op/s 2024-09-20T19:44:21.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:21 smithi160 ceph-mon[35859]: pgmap v226: 65 pgs: 65 active+clean; 56 MiB data, 349 MiB used, 536 GiB / 536 GiB avail; 4.8 MiB/s wr, 330 op/s 2024-09-20T19:44:23.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:23 smithi114 ceph-mon[28260]: pgmap v227: 65 pgs: 65 active+clean; 60 MiB data, 372 MiB used, 536 GiB / 536 GiB avail; 5.1 MiB/s wr, 352 op/s 2024-09-20T19:44:23.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:23 smithi160 ceph-mon[35859]: pgmap v227: 65 pgs: 65 active+clean; 60 MiB data, 372 MiB used, 536 GiB / 536 GiB avail; 5.1 MiB/s wr, 352 op/s 2024-09-20T19:44:25.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:25 smithi114 ceph-mon[28260]: pgmap v228: 65 pgs: 65 active+clean; 66 MiB data, 405 MiB used, 536 GiB / 536 GiB avail; 5.7 MiB/s wr, 397 op/s 2024-09-20T19:44:25.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:25 smithi160 ceph-mon[35859]: pgmap v228: 65 pgs: 65 active+clean; 66 MiB data, 405 MiB used, 536 GiB / 536 GiB avail; 5.7 MiB/s wr, 397 op/s 2024-09-20T19:44:26.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:26 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:44:26.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:26 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:44:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:27 smithi114 ceph-mon[28260]: pgmap v229: 65 pgs: 65 active+clean; 75 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 5.4 MiB/s wr, 339 op/s 2024-09-20T19:44:27.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:27 smithi160 ceph-mon[35859]: pgmap v229: 65 pgs: 65 active+clean; 75 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 5.4 MiB/s wr, 339 op/s 2024-09-20T19:44:29.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:29 smithi114 ceph-mon[28260]: pgmap v230: 65 pgs: 65 active+clean; 75 MiB data, 456 MiB used, 536 GiB / 536 GiB avail; 4.3 MiB/s wr, 233 op/s 2024-09-20T19:44:29.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:29 smithi160 ceph-mon[35859]: pgmap v230: 65 pgs: 65 active+clean; 75 MiB data, 456 MiB used, 536 GiB / 536 GiB avail; 4.3 MiB/s wr, 233 op/s 2024-09-20T19:44:31.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:31 smithi114 ceph-mon[28260]: pgmap v231: 65 pgs: 65 active+clean; 82 MiB data, 468 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s wr, 128 op/s 2024-09-20T19:44:31.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:31 smithi160 ceph-mon[35859]: pgmap v231: 65 pgs: 65 active+clean; 82 MiB data, 468 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s wr, 128 op/s 2024-09-20T19:44:32.577 INFO:tasks.workunit.client.1.smithi160.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-09-20T19:44:32.577 INFO:tasks.workunit.client.1.smithi160.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-09-20T19:44:32.578 INFO:tasks.workunit.client.1.smithi160.stderr:+ make 2024-09-20T19:44:33.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:33 smithi114 ceph-mon[28260]: pgmap v232: 65 pgs: 65 active+clean; 87 MiB data, 502 MiB used, 536 GiB / 536 GiB avail; 2.8 MiB/s wr, 159 op/s 2024-09-20T19:44:33.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:33 smithi160 ceph-mon[35859]: pgmap v232: 65 pgs: 65 active+clean; 87 MiB data, 502 MiB used, 536 GiB / 536 GiB avail; 2.8 MiB/s wr, 159 op/s 2024-09-20T19:44:35.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:35 smithi114 ceph-mon[28260]: pgmap v233: 65 pgs: 65 active+clean; 103 MiB data, 541 MiB used, 536 GiB / 536 GiB avail; 3.8 MiB/s wr, 347 op/s 2024-09-20T19:44:35.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:35 smithi160 ceph-mon[35859]: pgmap v233: 65 pgs: 65 active+clean; 103 MiB data, 541 MiB used, 536 GiB / 536 GiB avail; 3.8 MiB/s wr, 347 op/s 2024-09-20T19:44:37.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:37 smithi114 ceph-mon[28260]: pgmap v234: 65 pgs: 65 active+clean; 127 MiB data, 631 MiB used, 536 GiB / 536 GiB avail; 5.4 MiB/s wr, 611 op/s 2024-09-20T19:44:37.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:37 smithi160 ceph-mon[35859]: pgmap v234: 65 pgs: 65 active+clean; 127 MiB data, 631 MiB used, 536 GiB / 536 GiB avail; 5.4 MiB/s wr, 611 op/s 2024-09-20T19:44:39.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:39 smithi114 ceph-mon[28260]: pgmap v235: 65 pgs: 65 active+clean; 145 MiB data, 751 MiB used, 536 GiB / 536 GiB avail; 6.4 MiB/s wr, 835 op/s 2024-09-20T19:44:39.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:39 smithi160 ceph-mon[35859]: pgmap v235: 65 pgs: 65 active+clean; 145 MiB data, 751 MiB used, 536 GiB / 536 GiB avail; 6.4 MiB/s wr, 835 op/s 2024-09-20T19:44:40.729 INFO:tasks.workunit.client.1.smithi160.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-09-20T19:44:40.986 INFO:tasks.workunit.client.1.smithi160.stderr:++ readlink -f fsstress 2024-09-20T19:44:40.986 INFO:tasks.workunit.client.1.smithi160.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-09-20T19:44:40.986 INFO:tasks.workunit.client.1.smithi160.stderr:+ popd 2024-09-20T19:44:40.987 INFO:tasks.workunit.client.1.smithi160.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-09-20T19:44:40.987 INFO:tasks.workunit.client.1.smithi160.stderr:+ popd 2024-09-20T19:44:40.987 INFO:tasks.workunit.client.1.smithi160.stdout:~/cephtest/mnt.1/client.1/tmp 2024-09-20T19:44:40.987 INFO:tasks.workunit.client.1.smithi160.stderr:++ mktemp -d -p . 2024-09-20T19:44:40.988 INFO:tasks.workunit.client.1.smithi160.stderr:+ T=./tmp.ulPpNlnq2d 2024-09-20T19:44:40.988 INFO:tasks.workunit.client.1.smithi160.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.ulPpNlnq2d -l 1 -n 1000 -p 10 -v 2024-09-20T19:44:40.990 INFO:tasks.workunit.client.1.smithi160.stdout:seed = 1726139330 2024-09-20T19:44:40.993 INFO:tasks.workunit.client.1.smithi160.stdout:0/0: fsync - no filename 2024-09-20T19:44:40.993 INFO:tasks.workunit.client.1.smithi160.stdout:0/1: dwrite - no filename 2024-09-20T19:44:40.993 INFO:tasks.workunit.client.1.smithi160.stdout:0/2: read - no filename 2024-09-20T19:44:40.994 INFO:tasks.workunit.client.1.smithi160.stdout:1/0: dread - no filename 2024-09-20T19:44:40.994 INFO:tasks.workunit.client.1.smithi160.stdout:1/1: truncate - no filename 2024-09-20T19:44:40.994 INFO:tasks.workunit.client.1.smithi160.stdout:1/2: read - no filename 2024-09-20T19:44:40.994 INFO:tasks.workunit.client.1.smithi160.stdout:1/3: rename - no filename 2024-09-20T19:44:40.995 INFO:tasks.workunit.client.1.smithi160.stdout:0/3: creat f0 x:0 0 0 2024-09-20T19:44:40.997 INFO:tasks.workunit.client.1.smithi160.stdout:1/4: creat f0 x:0 0 0 2024-09-20T19:44:40.997 INFO:tasks.workunit.client.1.smithi160.stdout:1/5: write f0 [402723,60651] 0 2024-09-20T19:44:40.997 INFO:tasks.workunit.client.1.smithi160.stdout:1/6: truncate f0 821802 0 2024-09-20T19:44:40.997 INFO:tasks.workunit.client.1.smithi160.stdout:1/7: chown f0 61 1 2024-09-20T19:44:41.000 INFO:tasks.workunit.client.1.smithi160.stdout:2/0: creat f0 x:0 0 0 2024-09-20T19:44:41.003 INFO:tasks.workunit.client.1.smithi160.stdout:2/1: dread - f0 zero size 2024-09-20T19:44:41.004 INFO:tasks.workunit.client.1.smithi160.stdout:2/2: stat f0 0 2024-09-20T19:44:41.004 INFO:tasks.workunit.client.1.smithi160.stdout:2/3: chown f0 23513 1 2024-09-20T19:44:41.004 INFO:tasks.workunit.client.1.smithi160.stdout:4/0: dread - no filename 2024-09-20T19:44:41.004 INFO:tasks.workunit.client.1.smithi160.stdout:4/1: getdents . 0 2024-09-20T19:44:41.004 INFO:tasks.workunit.client.1.smithi160.stdout:4/2: fsync - no filename 2024-09-20T19:44:41.004 INFO:tasks.workunit.client.1.smithi160.stdout:4/3: chown . 124 1 2024-09-20T19:44:41.004 INFO:tasks.workunit.client.1.smithi160.stdout:4/4: fsync - no filename 2024-09-20T19:44:41.004 INFO:tasks.workunit.client.1.smithi160.stdout:4/5: write - no filename 2024-09-20T19:44:41.004 INFO:tasks.workunit.client.1.smithi160.stdout:4/6: write - no filename 2024-09-20T19:44:41.004 INFO:tasks.workunit.client.1.smithi160.stdout:4/7: chown . 65386 1 2024-09-20T19:44:41.004 INFO:tasks.workunit.client.1.smithi160.stdout:5/0: readlink - no filename 2024-09-20T19:44:41.004 INFO:tasks.workunit.client.1.smithi160.stdout:5/1: dwrite - no filename 2024-09-20T19:44:41.004 INFO:tasks.workunit.client.1.smithi160.stdout:5/2: dread - no filename 2024-09-20T19:44:41.005 INFO:tasks.workunit.client.1.smithi160.stdout:5/3: chown . 3012726 1 2024-09-20T19:44:41.005 INFO:tasks.workunit.client.1.smithi160.stdout:5/4: truncate - no filename 2024-09-20T19:44:41.006 INFO:tasks.workunit.client.1.smithi160.stdout:3/0: mkdir d0 0 2024-09-20T19:44:41.006 INFO:tasks.workunit.client.1.smithi160.stdout:3/1: write - no filename 2024-09-20T19:44:41.008 INFO:tasks.workunit.client.1.smithi160.stdout:5/5: creat f0 x:0 0 0 2024-09-20T19:44:41.009 INFO:tasks.workunit.client.1.smithi160.stdout:3/2: mknod d0/c1 0 2024-09-20T19:44:41.010 INFO:tasks.workunit.client.1.smithi160.stdout:7/0: rename - no filename 2024-09-20T19:44:41.011 INFO:tasks.workunit.client.1.smithi160.stdout:7/1: getdents . 0 2024-09-20T19:44:41.012 INFO:tasks.workunit.client.1.smithi160.stdout:6/0: mkdir d0 0 2024-09-20T19:44:41.012 INFO:tasks.workunit.client.1.smithi160.stdout:6/1: chown d0 25072 1 2024-09-20T19:44:41.012 INFO:tasks.workunit.client.1.smithi160.stdout:6/2: dread - no filename 2024-09-20T19:44:41.012 INFO:tasks.workunit.client.1.smithi160.stdout:6/3: dread - no filename 2024-09-20T19:44:41.012 INFO:tasks.workunit.client.1.smithi160.stdout:6/4: dwrite - no filename 2024-09-20T19:44:41.012 INFO:tasks.workunit.client.1.smithi160.stdout:6/5: read - no filename 2024-09-20T19:44:41.013 INFO:tasks.workunit.client.1.smithi160.stdout:3/3: creat d0/f2 x:0 0 0 2024-09-20T19:44:41.013 INFO:tasks.workunit.client.1.smithi160.stdout:0/4: dwrite f0 [0,4194304] 0 2024-09-20T19:44:41.014 INFO:tasks.workunit.client.1.smithi160.stdout:0/5: chown f0 0 1 2024-09-20T19:44:41.015 INFO:tasks.workunit.client.1.smithi160.stdout:8/0: dwrite - no filename 2024-09-20T19:44:41.015 INFO:tasks.workunit.client.1.smithi160.stdout:8/1: chown . 163738405 1 2024-09-20T19:44:41.015 INFO:tasks.workunit.client.1.smithi160.stdout:8/2: rename - no filename 2024-09-20T19:44:41.015 INFO:tasks.workunit.client.1.smithi160.stdout:8/3: unlink - no file 2024-09-20T19:44:41.015 INFO:tasks.workunit.client.1.smithi160.stdout:8/4: link - no file 2024-09-20T19:44:41.015 INFO:tasks.workunit.client.1.smithi160.stdout:8/5: rmdir - no directory 2024-09-20T19:44:41.016 INFO:tasks.workunit.client.1.smithi160.stdout:7/2: creat f0 x:0 0 0 2024-09-20T19:44:41.016 INFO:tasks.workunit.client.1.smithi160.stdout:7/3: fdatasync f0 0 2024-09-20T19:44:41.016 INFO:tasks.workunit.client.1.smithi160.stdout:7/4: readlink - no filename 2024-09-20T19:44:41.017 INFO:tasks.workunit.client.1.smithi160.stdout:6/6: rmdir d0 0 2024-09-20T19:44:41.017 INFO:tasks.workunit.client.1.smithi160.stdout:6/7: read - no filename 2024-09-20T19:44:41.018 INFO:tasks.workunit.client.1.smithi160.stdout:3/4: symlink d0/l3 0 2024-09-20T19:44:41.022 INFO:tasks.workunit.client.1.smithi160.stdout:0/6: creat f1 x:0 0 0 2024-09-20T19:44:41.023 INFO:tasks.workunit.client.1.smithi160.stdout:9/0: rename - no filename 2024-09-20T19:44:41.024 INFO:tasks.workunit.client.1.smithi160.stdout:6/8: creat f1 x:0 0 0 2024-09-20T19:44:41.026 INFO:tasks.workunit.client.1.smithi160.stdout:3/5: mknod d0/c4 0 2024-09-20T19:44:41.027 INFO:tasks.workunit.client.1.smithi160.stdout:9/1: mkdir d0 0 2024-09-20T19:44:41.028 INFO:tasks.workunit.client.1.smithi160.stdout:9/2: read - no filename 2024-09-20T19:44:41.028 INFO:tasks.workunit.client.1.smithi160.stdout:6/9: link f1 f2 0 2024-09-20T19:44:41.030 INFO:tasks.workunit.client.1.smithi160.stdout:9/3: creat d0/f1 x:0 0 0 2024-09-20T19:44:41.031 INFO:tasks.workunit.client.1.smithi160.stdout:6/10: symlink l3 0 2024-09-20T19:44:41.031 INFO:tasks.workunit.client.1.smithi160.stdout:5/6: dwrite f0 [0,4194304] 0 2024-09-20T19:44:41.032 INFO:tasks.workunit.client.1.smithi160.stdout:9/4: creat d0/f2 x:0 0 0 2024-09-20T19:44:41.032 INFO:tasks.workunit.client.1.smithi160.stdout:9/5: readlink - no filename 2024-09-20T19:44:41.034 INFO:tasks.workunit.client.1.smithi160.stdout:6/11: link f1 f4 0 2024-09-20T19:44:41.037 INFO:tasks.workunit.client.1.smithi160.stdout:9/6: unlink d0/f1 0 2024-09-20T19:44:41.038 INFO:tasks.workunit.client.1.smithi160.stdout:5/7: dread f0 [0,4194304] 0 2024-09-20T19:44:41.038 INFO:tasks.workunit.client.1.smithi160.stdout:6/12: link f2 f5 0 2024-09-20T19:44:41.039 INFO:tasks.workunit.client.1.smithi160.stdout:9/7: symlink d0/l3 0 2024-09-20T19:44:41.041 INFO:tasks.workunit.client.1.smithi160.stdout:5/8: write f0 [1595990,14286] 0 2024-09-20T19:44:41.042 INFO:tasks.workunit.client.1.smithi160.stdout:5/9: mkdir d1 0 2024-09-20T19:44:41.043 INFO:tasks.workunit.client.1.smithi160.stdout:5/10: chown f0 732904 1 2024-09-20T19:44:41.043 INFO:tasks.workunit.client.1.smithi160.stdout:5/11: mkdir d1/d2 0 2024-09-20T19:44:41.043 INFO:tasks.workunit.client.1.smithi160.stdout:5/12: readlink - no filename 2024-09-20T19:44:41.044 INFO:tasks.workunit.client.1.smithi160.stdout:5/13: creat d1/f3 x:0 0 0 2024-09-20T19:44:41.044 INFO:tasks.workunit.client.1.smithi160.stdout:5/14: chown d1/d2 150554 1 2024-09-20T19:44:41.044 INFO:tasks.workunit.client.1.smithi160.stdout:5/15: chown f0 0 1 2024-09-20T19:44:41.045 INFO:tasks.workunit.client.1.smithi160.stdout:5/16: creat d1/f4 x:0 0 0 2024-09-20T19:44:41.046 INFO:tasks.workunit.client.1.smithi160.stdout:5/17: creat d1/f5 x:0 0 0 2024-09-20T19:44:41.134 INFO:tasks.workunit.client.1.smithi160.stdout:2/4: fsync f0 0 2024-09-20T19:44:41.135 INFO:tasks.workunit.client.1.smithi160.stdout:2/5: rename f0 to f1 0 2024-09-20T19:44:41.143 INFO:tasks.workunit.client.1.smithi160.stdout:3/6: fsync d0/f2 0 2024-09-20T19:44:41.145 INFO:tasks.workunit.client.1.smithi160.stdout:7/5: fsync f0 0 2024-09-20T19:44:41.148 INFO:tasks.workunit.client.1.smithi160.stdout:7/6: creat f1 x:0 0 0 2024-09-20T19:44:41.149 INFO:tasks.workunit.client.1.smithi160.stdout:7/7: symlink l2 0 2024-09-20T19:44:41.150 INFO:tasks.workunit.client.1.smithi160.stdout:0/7: getdents . 0 2024-09-20T19:44:41.151 INFO:tasks.workunit.client.1.smithi160.stdout:0/8: readlink - no filename 2024-09-20T19:44:41.151 INFO:tasks.workunit.client.1.smithi160.stdout:7/8: mkdir d3 0 2024-09-20T19:44:41.151 INFO:tasks.workunit.client.1.smithi160.stdout:2/6: dwrite f1 [0,4194304] 0 2024-09-20T19:44:41.153 INFO:tasks.workunit.client.1.smithi160.stdout:0/9: rename f1 to f2 0 2024-09-20T19:44:41.158 INFO:tasks.workunit.client.1.smithi160.stdout:7/9: mkdir d3/d4 0 2024-09-20T19:44:41.160 INFO:tasks.workunit.client.1.smithi160.stdout:7/10: symlink d3/l5 0 2024-09-20T19:44:41.162 INFO:tasks.workunit.client.1.smithi160.stdout:7/11: symlink d3/l6 0 2024-09-20T19:44:41.162 INFO:tasks.workunit.client.1.smithi160.stdout:7/12: dread - f0 zero size 2024-09-20T19:44:41.162 INFO:tasks.workunit.client.1.smithi160.stdout:7/13: dread - f0 zero size 2024-09-20T19:44:41.164 INFO:tasks.workunit.client.1.smithi160.stdout:6/13: getdents . 0 2024-09-20T19:44:41.164 INFO:tasks.workunit.client.1.smithi160.stdout:6/14: dread - f5 zero size 2024-09-20T19:44:41.164 INFO:tasks.workunit.client.1.smithi160.stdout:6/15: readlink l3 0 2024-09-20T19:44:41.164 INFO:tasks.workunit.client.1.smithi160.stdout:6/16: rmdir - no directory 2024-09-20T19:44:41.165 INFO:tasks.workunit.client.1.smithi160.stdout:7/14: mknod d3/c7 0 2024-09-20T19:44:41.165 INFO:tasks.workunit.client.1.smithi160.stdout:9/8: getdents d0 0 2024-09-20T19:44:41.167 INFO:tasks.workunit.client.1.smithi160.stdout:6/17: mknod c6 0 2024-09-20T19:44:41.168 INFO:tasks.workunit.client.1.smithi160.stdout:7/15: creat d3/d4/f8 x:0 0 0 2024-09-20T19:44:41.168 INFO:tasks.workunit.client.1.smithi160.stdout:7/16: read - f0 zero size 2024-09-20T19:44:41.168 INFO:tasks.workunit.client.1.smithi160.stdout:2/7: dwrite f1 [0,4194304] 0 2024-09-20T19:44:41.170 INFO:tasks.workunit.client.1.smithi160.stdout:9/9: mkdir d0/d4 0 2024-09-20T19:44:41.172 INFO:tasks.workunit.client.1.smithi160.stdout:6/18: creat f7 x:0 0 0 2024-09-20T19:44:41.172 INFO:tasks.workunit.client.1.smithi160.stdout:0/10: write f0 [4170933,8004] 0 2024-09-20T19:44:41.174 INFO:tasks.workunit.client.1.smithi160.stdout:7/17: mkdir d3/d9 0 2024-09-20T19:44:41.174 INFO:tasks.workunit.client.1.smithi160.stdout:9/10: readlink d0/l3 0 2024-09-20T19:44:41.176 INFO:tasks.workunit.client.1.smithi160.stdout:2/8: write f1 [2932285,3900] 0 2024-09-20T19:44:41.176 INFO:tasks.workunit.client.1.smithi160.stdout:2/9: write f1 [898254,68774] 0 2024-09-20T19:44:41.176 INFO:tasks.workunit.client.1.smithi160.stdout:2/10: rmdir - no directory 2024-09-20T19:44:41.180 INFO:tasks.workunit.client.1.smithi160.stdout:6/19: mkdir d8 0 2024-09-20T19:44:41.183 INFO:tasks.workunit.client.1.smithi160.stdout:2/11: creat f2 x:0 0 0 2024-09-20T19:44:41.185 INFO:tasks.workunit.client.1.smithi160.stdout:0/11: link f0 f3 0 2024-09-20T19:44:41.185 INFO:tasks.workunit.client.1.smithi160.stdout:2/12: fdatasync f1 0 2024-09-20T19:44:41.185 INFO:tasks.workunit.client.1.smithi160.stdout:2/13: read - f2 zero size 2024-09-20T19:44:41.186 INFO:tasks.workunit.client.1.smithi160.stdout:2/14: dread - f2 zero size 2024-09-20T19:44:41.188 INFO:tasks.workunit.client.1.smithi160.stdout:6/20: mkdir d8/d9 0 2024-09-20T19:44:41.191 INFO:tasks.workunit.client.1.smithi160.stdout:2/15: mkdir d3 0 2024-09-20T19:44:41.194 INFO:tasks.workunit.client.1.smithi160.stdout:0/12: mkdir d4 0 2024-09-20T19:44:41.195 INFO:tasks.workunit.client.1.smithi160.stdout:2/16: rmdir d3 0 2024-09-20T19:44:41.195 INFO:tasks.workunit.client.1.smithi160.stdout:9/11: dwrite d0/f2 [0,4194304] 0 2024-09-20T19:44:41.197 INFO:tasks.workunit.client.1.smithi160.stdout:6/21: link f5 d8/d9/fa 0 2024-09-20T19:44:41.198 INFO:tasks.workunit.client.1.smithi160.stdout:0/13: creat d4/f5 x:0 0 0 2024-09-20T19:44:41.203 INFO:tasks.workunit.client.1.smithi160.stdout:2/17: unlink f1 0 2024-09-20T19:44:41.203 INFO:tasks.workunit.client.1.smithi160.stdout:2/18: write f2 [608570,7060] 0 2024-09-20T19:44:41.203 INFO:tasks.workunit.client.1.smithi160.stdout:2/19: rmdir - no directory 2024-09-20T19:44:41.208 INFO:tasks.workunit.client.1.smithi160.stdout:0/14: mkdir d4/d6 0 2024-09-20T19:44:41.209 INFO:tasks.workunit.client.1.smithi160.stdout:2/20: creat f4 x:0 0 0 2024-09-20T19:44:41.209 INFO:tasks.workunit.client.1.smithi160.stdout:2/21: dread - f4 zero size 2024-09-20T19:44:41.209 INFO:tasks.workunit.client.1.smithi160.stdout:2/22: stat f2 0 2024-09-20T19:44:41.212 INFO:tasks.workunit.client.1.smithi160.stdout:9/12: dwrite d0/f2 [0,4194304] 0 2024-09-20T19:44:41.212 INFO:tasks.workunit.client.1.smithi160.stdout:6/22: rename l3 to d8/lb 0 2024-09-20T19:44:41.212 INFO:tasks.workunit.client.1.smithi160.stdout:2/23: rename f4 to f5 0 2024-09-20T19:44:41.213 INFO:tasks.workunit.client.1.smithi160.stdout:0/15: dread f0 [0,4194304] 0 2024-09-20T19:44:41.214 INFO:tasks.workunit.client.1.smithi160.stdout:0/16: dread - f2 zero size 2024-09-20T19:44:41.214 INFO:tasks.workunit.client.1.smithi160.stdout:0/17: chown d4/d6 5 1 2024-09-20T19:44:41.214 INFO:tasks.workunit.client.1.smithi160.stdout:9/13: mkdir d0/d4/d5 0 2024-09-20T19:44:41.215 INFO:tasks.workunit.client.1.smithi160.stdout:6/23: fdatasync f4 0 2024-09-20T19:44:41.217 INFO:tasks.workunit.client.1.smithi160.stdout:2/24: creat f6 x:0 0 0 2024-09-20T19:44:41.218 INFO:tasks.workunit.client.1.smithi160.stdout:0/18: creat d4/f7 x:0 0 0 2024-09-20T19:44:41.218 INFO:tasks.workunit.client.1.smithi160.stdout:0/19: write f2 [686822,20526] 0 2024-09-20T19:44:41.218 INFO:tasks.workunit.client.1.smithi160.stdout:0/20: getdents d4/d6 0 2024-09-20T19:44:41.222 INFO:tasks.workunit.client.1.smithi160.stdout:6/24: rename f7 to d8/d9/fc 0 2024-09-20T19:44:41.223 INFO:tasks.workunit.client.1.smithi160.stdout:6/25: truncate d8/d9/fc 549061 0 2024-09-20T19:44:41.223 INFO:tasks.workunit.client.1.smithi160.stdout:6/26: dread - d8/d9/fa zero size 2024-09-20T19:44:41.225 INFO:tasks.workunit.client.1.smithi160.stdout:0/21: link f3 d4/f8 0 2024-09-20T19:44:41.225 INFO:tasks.workunit.client.1.smithi160.stdout:0/22: chown f2 5872875 1 2024-09-20T19:44:41.229 INFO:tasks.workunit.client.1.smithi160.stdout:9/14: symlink d0/l6 0 2024-09-20T19:44:41.229 INFO:tasks.workunit.client.1.smithi160.stdout:9/15: fsync d0/f2 0 2024-09-20T19:44:41.231 INFO:tasks.workunit.client.1.smithi160.stdout:6/27: link f4 d8/d9/fd 0 2024-09-20T19:44:41.265 INFO:tasks.workunit.client.1.smithi160.stdout:9/16: dwrite d0/f2 [0,4194304] 0 2024-09-20T19:44:41.265 INFO:tasks.workunit.client.1.smithi160.stdout:0/23: dwrite d4/f5 [0,4194304] 0 2024-09-20T19:44:41.265 INFO:tasks.workunit.client.1.smithi160.stdout:6/28: dwrite d8/d9/fc [0,4194304] 0 2024-09-20T19:44:41.269 INFO:tasks.workunit.client.1.smithi160.stdout:0/24: mknod d4/d6/c9 0 2024-09-20T19:44:41.274 INFO:tasks.workunit.client.1.smithi160.stdout:6/29: write f1 [259496,9861] 0 2024-09-20T19:44:41.276 INFO:tasks.workunit.client.1.smithi160.stdout:9/17: symlink d0/l7 0 2024-09-20T19:44:41.286 INFO:tasks.workunit.client.1.smithi160.stdout:9/18: write d0/f2 [3983352,19107] 0 2024-09-20T19:44:41.291 INFO:tasks.workunit.client.1.smithi160.stdout:0/25: dwrite d4/f7 [0,4194304] 0 2024-09-20T19:44:41.299 INFO:tasks.workunit.client.1.smithi160.stdout:0/26: write d4/f7 [684091,103813] 0 2024-09-20T19:44:41.308 INFO:tasks.workunit.client.1.smithi160.stdout:9/19: dwrite d0/f2 [0,4194304] 0 2024-09-20T19:44:41.308 INFO:tasks.workunit.client.1.smithi160.stdout:9/20: chown d0/f2 517 1 2024-09-20T19:44:41.325 INFO:tasks.workunit.client.1.smithi160.stdout:9/21: dwrite d0/f2 [0,4194304] 0 2024-09-20T19:44:41.334 INFO:tasks.workunit.client.1.smithi160.stdout:9/22: dread d0/f2 [0,4194304] 0 2024-09-20T19:44:41.339 INFO:tasks.workunit.client.1.smithi160.stdout:9/23: write d0/f2 [880296,52039] 0 2024-09-20T19:44:41.339 INFO:tasks.workunit.client.1.smithi160.stdout:9/24: getdents d0/d4/d5 0 2024-09-20T19:44:41.349 INFO:tasks.workunit.client.1.smithi160.stdout:9/25: dread d0/f2 [0,4194304] 0 2024-09-20T19:44:41.351 INFO:tasks.workunit.client.1.smithi160.stdout:9/26: mkdir d0/d8 0 2024-09-20T19:44:41.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:41 smithi114 ceph-mon[28260]: pgmap v236: 65 pgs: 65 active+clean; 165 MiB data, 825 MiB used, 536 GiB / 536 GiB avail; 8.0 MiB/s wr, 965 op/s 2024-09-20T19:44:41.754 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:41 smithi114 ceph-mon[28260]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:44:41.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:41 smithi160 ceph-mon[35859]: pgmap v236: 65 pgs: 65 active+clean; 165 MiB data, 825 MiB used, 536 GiB / 536 GiB avail; 8.0 MiB/s wr, 965 op/s 2024-09-20T19:44:41.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:41 smithi160 ceph-mon[35859]: from='mgr.14219 172.21.15.114:0/2385672768' entity='mgr.smithi114.rvhgji' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T19:44:43.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:43 smithi114 ceph-mon[28260]: pgmap v237: 65 pgs: 65 active+clean; 170 MiB data, 841 MiB used, 536 GiB / 536 GiB avail; 341 KiB/s rd, 7.8 MiB/s wr, 960 op/s 2024-09-20T19:44:43.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:43 smithi160 ceph-mon[35859]: pgmap v237: 65 pgs: 65 active+clean; 170 MiB data, 841 MiB used, 536 GiB / 536 GiB avail; 341 KiB/s rd, 7.8 MiB/s wr, 960 op/s 2024-09-20T19:44:44.071 INFO:tasks.workunit.client.1.smithi160.stdout:9/27: sync 2024-09-20T19:44:44.071 INFO:tasks.workunit.client.1.smithi160.stdout:1/8: sync 2024-09-20T19:44:44.071 INFO:tasks.workunit.client.1.smithi160.stdout:4/8: sync 2024-09-20T19:44:44.071 INFO:tasks.workunit.client.1.smithi160.stdout:7/18: sync 2024-09-20T19:44:44.071 INFO:tasks.workunit.client.1.smithi160.stdout:4/9: write - no filename 2024-09-20T19:44:44.071 INFO:tasks.workunit.client.1.smithi160.stdout:4/10: stat - no entries 2024-09-20T19:44:44.071 INFO:tasks.workunit.client.1.smithi160.stdout:6/30: sync 2024-09-20T19:44:44.071 INFO:tasks.workunit.client.1.smithi160.stdout:4/11: readlink - no filename 2024-09-20T19:44:44.071 INFO:tasks.workunit.client.1.smithi160.stdout:3/7: sync 2024-09-20T19:44:44.072 INFO:tasks.workunit.client.1.smithi160.stdout:7/19: write f1 [828854,47537] 0 2024-09-20T19:44:44.072 INFO:tasks.workunit.client.1.smithi160.stdout:3/8: write d0/f2 [797479,95882] 0 2024-09-20T19:44:44.072 INFO:tasks.workunit.client.1.smithi160.stdout:8/6: sync 2024-09-20T19:44:44.072 INFO:tasks.workunit.client.1.smithi160.stdout:2/25: sync 2024-09-20T19:44:44.072 INFO:tasks.workunit.client.1.smithi160.stdout:5/18: sync 2024-09-20T19:44:44.072 INFO:tasks.workunit.client.1.smithi160.stdout:0/27: sync 2024-09-20T19:44:44.072 INFO:tasks.workunit.client.1.smithi160.stdout:8/7: readlink - no filename 2024-09-20T19:44:44.072 INFO:tasks.workunit.client.1.smithi160.stdout:8/8: rename - no filename 2024-09-20T19:44:44.072 INFO:tasks.workunit.client.1.smithi160.stdout:8/9: chown . 175 1 2024-09-20T19:44:44.072 INFO:tasks.workunit.client.1.smithi160.stdout:8/10: dread - no filename 2024-09-20T19:44:44.072 INFO:tasks.workunit.client.1.smithi160.stdout:8/11: rename - no filename 2024-09-20T19:44:44.072 INFO:tasks.workunit.client.1.smithi160.stdout:9/28: symlink d0/d8/l9 0 2024-09-20T19:44:44.073 INFO:tasks.workunit.client.1.smithi160.stdout:9/29: readlink d0/l7 0 2024-09-20T19:44:44.073 INFO:tasks.workunit.client.1.smithi160.stdout:1/9: symlink l1 0 2024-09-20T19:44:44.073 INFO:tasks.workunit.client.1.smithi160.stdout:1/10: chown l1 42 1 2024-09-20T19:44:44.074 INFO:tasks.workunit.client.1.smithi160.stdout:4/12: mkdir d0 0 2024-09-20T19:44:44.074 INFO:tasks.workunit.client.1.smithi160.stdout:1/11: read f0 [418437,93757] 0 2024-09-20T19:44:44.075 INFO:tasks.workunit.client.1.smithi160.stdout:3/9: rmdir d0 39 2024-09-20T19:44:44.077 INFO:tasks.workunit.client.1.smithi160.stdout:5/19: mknod d1/d2/c6 0 2024-09-20T19:44:44.079 INFO:tasks.workunit.client.1.smithi160.stdout:8/12: creat f0 x:0 0 0 2024-09-20T19:44:44.080 INFO:tasks.workunit.client.1.smithi160.stdout:4/13: creat d0/f1 x:0 0 0 2024-09-20T19:44:44.081 INFO:tasks.workunit.client.1.smithi160.stdout:5/20: mkdir d1/d7 0 2024-09-20T19:44:44.082 INFO:tasks.workunit.client.1.smithi160.stdout:0/28: write d4/f5 [2932902,68047] 0 2024-09-20T19:44:44.083 INFO:tasks.workunit.client.1.smithi160.stdout:4/14: rmdir d0 39 2024-09-20T19:44:44.083 INFO:tasks.workunit.client.1.smithi160.stdout:4/15: fdatasync d0/f1 0 2024-09-20T19:44:44.085 INFO:tasks.workunit.client.1.smithi160.stdout:5/21: mkdir d1/d7/d8 0 2024-09-20T19:44:44.091 INFO:tasks.workunit.client.1.smithi160.stdout:0/29: truncate f0 2577926 0 2024-09-20T19:44:44.091 INFO:tasks.workunit.client.1.smithi160.stdout:0/30: fdatasync f0 0 2024-09-20T19:44:44.092 INFO:tasks.workunit.client.1.smithi160.stdout:4/16: mkdir d0/d2 0 2024-09-20T19:44:44.094 INFO:tasks.workunit.client.1.smithi160.stdout:1/12: write f0 [239559,91960] 0 2024-09-20T19:44:44.095 INFO:tasks.workunit.client.1.smithi160.stdout:1/13: chown f0 3162 1 2024-09-20T19:44:44.096 INFO:tasks.workunit.client.1.smithi160.stdout:5/22: rename d1/f3 to d1/d7/f9 0 2024-09-20T19:44:44.104 INFO:tasks.workunit.client.1.smithi160.stdout:3/10: dread d0/f2 [0,4194304] 0 2024-09-20T19:44:44.107 INFO:tasks.workunit.client.1.smithi160.stdout:1/14: write f0 [718219,26884] 0 2024-09-20T19:44:44.107 INFO:tasks.workunit.client.1.smithi160.stdout:3/11: dread d0/f2 [0,4194304] 0 2024-09-20T19:44:44.107 INFO:tasks.workunit.client.1.smithi160.stdout:3/12: chown d0/c1 15 1 2024-09-20T19:44:44.110 INFO:tasks.workunit.client.1.smithi160.stdout:3/13: write d0/f2 [279188,109352] 0 2024-09-20T19:44:44.110 INFO:tasks.workunit.client.1.smithi160.stdout:3/14: chown d0/l3 31008 1 2024-09-20T19:44:44.110 INFO:tasks.workunit.client.1.smithi160.stdout:3/15: stat d0/c4 0 2024-09-20T19:44:44.114 INFO:tasks.workunit.client.1.smithi160.stdout:7/20: dwrite d3/d4/f8 [0,4194304] 0 2024-09-20T19:44:44.117 INFO:tasks.workunit.client.1.smithi160.stdout:7/21: mknod d3/d9/ca 0 2024-09-20T19:44:44.128 INFO:tasks.workunit.client.1.smithi160.stdout:2/26: dwrite f2 [0,4194304] 0 2024-09-20T19:44:44.128 INFO:tasks.workunit.client.1.smithi160.stdout:2/27: dread - f6 zero size 2024-09-20T19:44:44.128 INFO:tasks.workunit.client.1.smithi160.stdout:2/28: dread - f5 zero size 2024-09-20T19:44:44.129 INFO:tasks.workunit.client.1.smithi160.stdout:6/31: dwrite f5 [0,4194304] 0 2024-09-20T19:44:44.129 INFO:tasks.workunit.client.1.smithi160.stdout:8/13: dwrite f0 [0,4194304] 0 2024-09-20T19:44:44.129 INFO:tasks.workunit.client.1.smithi160.stdout:4/17: dwrite d0/f1 [0,4194304] 0 2024-09-20T19:44:44.130 INFO:tasks.workunit.client.1.smithi160.stdout:9/30: dwrite d0/f2 [0,4194304] 0 2024-09-20T19:44:44.133 INFO:tasks.workunit.client.1.smithi160.stdout:0/31: dread d4/f5 [0,4194304] 0 2024-09-20T19:44:44.133 INFO:tasks.workunit.client.1.smithi160.stdout:7/22: dread d3/d4/f8 [0,4194304] 0 2024-09-20T19:44:44.134 INFO:tasks.workunit.client.1.smithi160.stdout:6/32: rmdir d8/d9 39 2024-09-20T19:44:44.134 INFO:tasks.workunit.client.1.smithi160.stdout:6/33: chown d8/d9 46580 1 2024-09-20T19:44:44.140 INFO:tasks.workunit.client.1.smithi160.stdout:8/14: mkdir d1 0 2024-09-20T19:44:44.142 INFO:tasks.workunit.client.1.smithi160.stdout:4/18: creat d0/f3 x:0 0 0 2024-09-20T19:44:44.144 INFO:tasks.workunit.client.1.smithi160.stdout:0/32: creat d4/d6/fa x:0 0 0 2024-09-20T19:44:44.146 INFO:tasks.workunit.client.1.smithi160.stdout:8/15: mknod d1/c2 0 2024-09-20T19:44:44.146 INFO:tasks.workunit.client.1.smithi160.stdout:8/16: readlink - no filename 2024-09-20T19:44:44.146 INFO:tasks.workunit.client.1.smithi160.stdout:9/31: dread d0/f2 [0,4194304] 0 2024-09-20T19:44:44.147 INFO:tasks.workunit.client.1.smithi160.stdout:4/19: mkdir d0/d4 0 2024-09-20T19:44:44.148 INFO:tasks.workunit.client.1.smithi160.stdout:0/33: rmdir d4 39 2024-09-20T19:44:44.152 INFO:tasks.workunit.client.1.smithi160.stdout:4/20: unlink d0/f3 0 2024-09-20T19:44:44.152 INFO:tasks.workunit.client.1.smithi160.stdout:9/32: dread d0/f2 [0,4194304] 0 2024-09-20T19:44:44.152 INFO:tasks.workunit.client.1.smithi160.stdout:2/29: dwrite f2 [0,4194304] 0 2024-09-20T19:44:44.152 INFO:tasks.workunit.client.1.smithi160.stdout:9/33: fsync d0/f2 0 2024-09-20T19:44:44.152 INFO:tasks.workunit.client.1.smithi160.stdout:2/30: dread - f6 zero size 2024-09-20T19:44:44.152 INFO:tasks.workunit.client.1.smithi160.stdout:9/34: stat d0/l7 0 2024-09-20T19:44:44.153 INFO:tasks.workunit.client.1.smithi160.stdout:0/34: mkdir d4/d6/db 0 2024-09-20T19:44:44.156 INFO:tasks.workunit.client.1.smithi160.stdout:9/35: creat d0/fa x:0 0 0 2024-09-20T19:44:44.158 INFO:tasks.workunit.client.1.smithi160.stdout:8/17: dread f0 [0,4194304] 0 2024-09-20T19:44:44.160 INFO:tasks.workunit.client.1.smithi160.stdout:8/18: creat d1/f3 x:0 0 0 2024-09-20T19:44:44.160 INFO:tasks.workunit.client.1.smithi160.stdout:8/19: dread - d1/f3 zero size 2024-09-20T19:44:44.161 INFO:tasks.workunit.client.1.smithi160.stdout:8/20: creat d1/f4 x:0 0 0 2024-09-20T19:44:44.161 INFO:tasks.workunit.client.1.smithi160.stdout:8/21: write d1/f4 [854360,24272] 0 2024-09-20T19:44:44.161 INFO:tasks.workunit.client.1.smithi160.stdout:8/22: chown d1/f4 915 1 2024-09-20T19:44:44.163 INFO:tasks.workunit.client.1.smithi160.stdout:4/21: write d0/f1 [385294,49979] 0 2024-09-20T19:44:44.163 INFO:tasks.workunit.client.1.smithi160.stdout:4/22: readlink - no filename 2024-09-20T19:44:44.167 INFO:tasks.workunit.client.1.smithi160.stdout:4/23: link d0/f1 d0/d4/f5 0 2024-09-20T19:44:44.174 INFO:tasks.workunit.client.1.smithi160.stdout:0/35: dwrite d4/f7 [0,4194304] 0 2024-09-20T19:44:44.176 INFO:tasks.workunit.client.1.smithi160.stdout:0/36: dread f2 [0,4194304] 0 2024-09-20T19:44:44.349 INFO:tasks.workunit.client.1.smithi160.stdout:5/23: rename d1/d2 to d1/d7/da 0 2024-09-20T19:44:44.353 INFO:tasks.workunit.client.1.smithi160.stdout:5/24: dread f0 [0,4194304] 0 2024-09-20T19:44:44.356 INFO:tasks.workunit.client.1.smithi160.stdout:0/37: fsync d4/f8 0 2024-09-20T19:44:44.362 INFO:tasks.workunit.client.1.smithi160.stdout:1/15: truncate f0 71159 0 2024-09-20T19:44:44.364 INFO:tasks.workunit.client.1.smithi160.stdout:3/16: truncate d0/f2 708462 0 2024-09-20T19:44:44.364 INFO:tasks.workunit.client.1.smithi160.stdout:1/16: mknod c2 0 2024-09-20T19:44:44.366 INFO:tasks.workunit.client.1.smithi160.stdout:3/17: mknod d0/c5 0 2024-09-20T19:44:44.371 INFO:tasks.workunit.client.1.smithi160.stdout:5/25: dwrite d1/f5 [0,4194304] 0 2024-09-20T19:44:44.371 INFO:tasks.workunit.client.1.smithi160.stdout:5/26: chown d1/d7/da 406685 1 2024-09-20T19:44:44.372 INFO:tasks.workunit.client.1.smithi160.stdout:4/24: write d0/f1 [4867087,124647] 0 2024-09-20T19:44:44.375 INFO:tasks.workunit.client.1.smithi160.stdout:5/27: unlink d1/d7/da/c6 0 2024-09-20T19:44:44.380 INFO:tasks.workunit.client.1.smithi160.stdout:5/28: creat d1/d7/fb x:0 0 0 2024-09-20T19:44:44.380 INFO:tasks.workunit.client.1.smithi160.stdout:5/29: fsync d1/d7/f9 0 2024-09-20T19:44:44.383 INFO:tasks.workunit.client.1.smithi160.stdout:5/30: link d1/f4 d1/d7/d8/fc 0 2024-09-20T19:44:44.383 INFO:tasks.workunit.client.1.smithi160.stdout:5/31: stat d1/d7/d8 0 2024-09-20T19:44:44.383 INFO:tasks.workunit.client.1.smithi160.stdout:5/32: chown f0 45256207 1 2024-09-20T19:44:44.383 INFO:tasks.workunit.client.1.smithi160.stdout:5/33: chown d1/d7/d8 1954 1 2024-09-20T19:44:44.383 INFO:tasks.workunit.client.1.smithi160.stdout:4/25: dread d0/f1 [0,4194304] 0 2024-09-20T19:44:44.385 INFO:tasks.workunit.client.1.smithi160.stdout:5/34: creat d1/d7/fd x:0 0 0 2024-09-20T19:44:44.385 INFO:tasks.workunit.client.1.smithi160.stdout:5/35: dread - d1/d7/fd zero size 2024-09-20T19:44:44.385 INFO:tasks.workunit.client.1.smithi160.stdout:5/36: stat d1/d7/fd 0 2024-09-20T19:44:44.387 INFO:tasks.workunit.client.1.smithi160.stdout:4/26: unlink d0/d4/f5 0 2024-09-20T19:44:44.388 INFO:tasks.workunit.client.1.smithi160.stdout:9/36: getdents d0 0 2024-09-20T19:44:44.389 INFO:tasks.workunit.client.1.smithi160.stdout:2/31: dwrite f2 [4194304,4194304] 0 2024-09-20T19:44:44.389 INFO:tasks.workunit.client.1.smithi160.stdout:6/34: dwrite f2 [4194304,4194304] 0 2024-09-20T19:44:44.389 INFO:tasks.workunit.client.1.smithi160.stdout:6/35: write f1 [8931231,9816] 0 2024-09-20T19:44:44.390 INFO:tasks.workunit.client.1.smithi160.stdout:4/27: mknod d0/d2/c6 0 2024-09-20T19:44:44.390 INFO:tasks.workunit.client.1.smithi160.stdout:8/23: fsync d1/f4 0 2024-09-20T19:44:44.390 INFO:tasks.workunit.client.1.smithi160.stdout:8/24: chown d1/c2 23366 1 2024-09-20T19:44:44.391 INFO:tasks.workunit.client.1.smithi160.stdout:9/37: creat d0/d4/fb x:0 0 0 2024-09-20T19:44:44.391 INFO:tasks.workunit.client.1.smithi160.stdout:9/38: dread - d0/fa zero size 2024-09-20T19:44:44.391 INFO:tasks.workunit.client.1.smithi160.stdout:9/39: truncate d0/fa 241438 0 2024-09-20T19:44:44.397 INFO:tasks.workunit.client.1.smithi160.stdout:6/36: rmdir d8 39 2024-09-20T19:44:44.398 INFO:tasks.workunit.client.1.smithi160.stdout:2/32: dread f2 [0,4194304] 0 2024-09-20T19:44:44.398 INFO:tasks.workunit.client.1.smithi160.stdout:2/33: chown f6 685 1 2024-09-20T19:44:44.398 INFO:tasks.workunit.client.1.smithi160.stdout:2/34: chown f2 965117 1 2024-09-20T19:44:44.401 INFO:tasks.workunit.client.1.smithi160.stdout:8/25: mknod d1/c5 0 2024-09-20T19:44:44.401 INFO:tasks.workunit.client.1.smithi160.stdout:8/26: dread - d1/f3 zero size 2024-09-20T19:44:44.403 INFO:tasks.workunit.client.1.smithi160.stdout:9/40: creat d0/fc x:0 0 0 2024-09-20T19:44:44.403 INFO:tasks.workunit.client.1.smithi160.stdout:9/41: fdatasync d0/f2 0 2024-09-20T19:44:44.404 INFO:tasks.workunit.client.1.smithi160.stdout:4/28: dread d0/f1 [0,4194304] 0 2024-09-20T19:44:44.404 INFO:tasks.workunit.client.1.smithi160.stdout:4/29: chown d0/d2/c6 3641858 1 2024-09-20T19:44:44.404 INFO:tasks.workunit.client.1.smithi160.stdout:5/37: dwrite d1/d7/fb [0,4194304] 0 2024-09-20T19:44:44.407 INFO:tasks.workunit.client.1.smithi160.stdout:2/35: mkdir d7 0 2024-09-20T19:44:44.407 INFO:tasks.workunit.client.1.smithi160.stdout:2/36: readlink - no filename 2024-09-20T19:44:44.415 INFO:tasks.workunit.client.1.smithi160.stdout:2/37: symlink d7/l8 0 2024-09-20T19:44:44.423 INFO:tasks.workunit.client.1.smithi160.stdout:2/38: rmdir d7 39 2024-09-20T19:44:44.423 INFO:tasks.workunit.client.1.smithi160.stdout:2/39: write f6 [499276,37620] 0 2024-09-20T19:44:44.423 INFO:tasks.workunit.client.1.smithi160.stdout:3/18: write d0/f2 [838454,1986] 0 2024-09-20T19:44:44.423 INFO:tasks.workunit.client.1.smithi160.stdout:3/19: readlink d0/l3 0 2024-09-20T19:44:44.424 INFO:tasks.workunit.client.1.smithi160.stdout:3/20: read d0/f2 [474295,91028] 0 2024-09-20T19:44:44.426 INFO:tasks.workunit.client.1.smithi160.stdout:2/40: mknod d7/c9 0 2024-09-20T19:44:44.427 INFO:tasks.workunit.client.1.smithi160.stdout:7/23: sync 2024-09-20T19:44:44.430 INFO:tasks.workunit.client.1.smithi160.stdout:3/21: mkdir d0/d6 0 2024-09-20T19:44:44.430 INFO:tasks.workunit.client.1.smithi160.stdout:3/22: write d0/f2 [1454671,40053] 0 2024-09-20T19:44:44.431 INFO:tasks.workunit.client.1.smithi160.stdout:2/41: rename f6 to d7/fa 0 2024-09-20T19:44:44.434 INFO:tasks.workunit.client.1.smithi160.stdout:3/23: mkdir d0/d7 0 2024-09-20T19:44:44.435 INFO:tasks.workunit.client.1.smithi160.stdout:8/27: dwrite d1/f3 [0,4194304] 0 2024-09-20T19:44:44.435 INFO:tasks.workunit.client.1.smithi160.stdout:8/28: chown d1/c5 150745 1 2024-09-20T19:44:44.435 INFO:tasks.workunit.client.1.smithi160.stdout:5/38: truncate d1/f5 2077687 0 2024-09-20T19:44:44.436 INFO:tasks.workunit.client.1.smithi160.stdout:3/24: mknod d0/d6/c8 0 2024-09-20T19:44:44.442 INFO:tasks.workunit.client.1.smithi160.stdout:5/39: creat d1/d7/d8/fe x:0 0 0 2024-09-20T19:44:44.444 INFO:tasks.workunit.client.1.smithi160.stdout:3/25: mkdir d0/d7/d9 0 2024-09-20T19:44:44.445 INFO:tasks.workunit.client.1.smithi160.stdout:3/26: write d0/f2 [244618,27068] 0 2024-09-20T19:44:44.447 INFO:tasks.workunit.client.1.smithi160.stdout:3/27: creat d0/d7/fa x:0 0 0 2024-09-20T19:44:44.447 INFO:tasks.workunit.client.1.smithi160.stdout:3/28: stat d0/l3 0 2024-09-20T19:44:44.447 INFO:tasks.workunit.client.1.smithi160.stdout:3/29: fdatasync d0/d7/fa 0 2024-09-20T19:44:44.447 INFO:tasks.workunit.client.1.smithi160.stdout:3/30: dread - d0/d7/fa zero size 2024-09-20T19:44:44.447 INFO:tasks.workunit.client.1.smithi160.stdout:3/31: write d0/f2 [1538491,14286] 0 2024-09-20T19:44:44.447 INFO:tasks.workunit.client.1.smithi160.stdout:3/32: write d0/d7/fa [370848,123711] 0 2024-09-20T19:44:44.447 INFO:tasks.workunit.client.1.smithi160.stdout:3/33: write d0/f2 [1835083,55977] 0 2024-09-20T19:44:44.448 INFO:tasks.workunit.client.1.smithi160.stdout:9/42: dwrite d0/d4/fb [0,4194304] 0 2024-09-20T19:44:44.448 INFO:tasks.workunit.client.1.smithi160.stdout:6/37: dwrite f2 [8388608,4194304] 0 2024-09-20T19:44:44.449 INFO:tasks.workunit.client.1.smithi160.stdout:4/30: dwrite d0/f1 [0,4194304] 0 2024-09-20T19:44:44.449 INFO:tasks.workunit.client.1.smithi160.stdout:4/31: readlink - no filename 2024-09-20T19:44:44.449 INFO:tasks.workunit.client.1.smithi160.stdout:2/42: write d7/fa [180635,53428] 0 2024-09-20T19:44:44.450 INFO:tasks.workunit.client.1.smithi160.stdout:3/34: mkdir d0/db 0 2024-09-20T19:44:44.459 INFO:tasks.workunit.client.1.smithi160.stdout:6/38: creat d8/d9/fe x:0 0 0 2024-09-20T19:44:44.464 INFO:tasks.workunit.client.1.smithi160.stdout:2/43: mknod d7/cb 0 2024-09-20T19:44:44.465 INFO:tasks.workunit.client.1.smithi160.stdout:3/35: mkdir d0/db/dc 0 2024-09-20T19:44:44.465 INFO:tasks.workunit.client.1.smithi160.stdout:3/36: write d0/d7/fa [647335,115465] 0 2024-09-20T19:44:44.465 INFO:tasks.workunit.client.1.smithi160.stdout:4/32: read d0/f1 [1190467,129844] 0 2024-09-20T19:44:44.465 INFO:tasks.workunit.client.1.smithi160.stdout:8/29: dwrite f0 [0,4194304] 0 2024-09-20T19:44:44.471 INFO:tasks.workunit.client.1.smithi160.stdout:4/33: dread d0/f1 [0,4194304] 0 2024-09-20T19:44:44.471 INFO:tasks.workunit.client.1.smithi160.stdout:4/34: chown d0/d2/c6 33 1 2024-09-20T19:44:44.471 INFO:tasks.workunit.client.1.smithi160.stdout:4/35: rename d0 to d0/d2/d7 22 2024-09-20T19:44:44.471 INFO:tasks.workunit.client.1.smithi160.stdout:4/36: truncate d0/f1 5874218 0 2024-09-20T19:44:44.471 INFO:tasks.workunit.client.1.smithi160.stdout:4/37: fdatasync d0/f1 0 2024-09-20T19:44:44.472 INFO:tasks.workunit.client.1.smithi160.stdout:4/38: stat d0/d2 0 2024-09-20T19:44:44.474 INFO:tasks.workunit.client.1.smithi160.stdout:6/39: symlink d8/lf 0 2024-09-20T19:44:44.474 INFO:tasks.workunit.client.1.smithi160.stdout:6/40: chown d8 119309440 1 2024-09-20T19:44:44.474 INFO:tasks.workunit.client.1.smithi160.stdout:9/43: dwrite d0/fc [0,4194304] 0 2024-09-20T19:44:44.475 INFO:tasks.workunit.client.1.smithi160.stdout:2/44: creat d7/fc x:0 0 0 2024-09-20T19:44:44.479 INFO:tasks.workunit.client.1.smithi160.stdout:3/37: symlink d0/ld 0 2024-09-20T19:44:44.479 INFO:tasks.workunit.client.1.smithi160.stdout:2/45: dread d7/fa [0,4194304] 0 2024-09-20T19:44:44.483 INFO:tasks.workunit.client.1.smithi160.stdout:8/30: mknod d1/c6 0 2024-09-20T19:44:44.484 INFO:tasks.workunit.client.1.smithi160.stdout:4/39: symlink d0/d2/l8 0 2024-09-20T19:44:44.492 INFO:tasks.workunit.client.1.smithi160.stdout:6/41: dwrite d8/d9/fc [0,4194304] 0 2024-09-20T19:44:44.495 INFO:tasks.workunit.client.1.smithi160.stdout:4/40: mkdir d0/d2/d9 0 2024-09-20T19:44:44.500 INFO:tasks.workunit.client.1.smithi160.stdout:8/31: dread f0 [0,4194304] 0 2024-09-20T19:44:44.503 INFO:tasks.workunit.client.1.smithi160.stdout:8/32: dread d1/f4 [0,4194304] 0 2024-09-20T19:44:44.504 INFO:tasks.workunit.client.1.smithi160.stdout:4/41: dread d0/f1 [0,4194304] 0 2024-09-20T19:44:44.508 INFO:tasks.workunit.client.1.smithi160.stdout:2/46: dwrite f2 [4194304,4194304] 0 2024-09-20T19:44:44.508 INFO:tasks.workunit.client.1.smithi160.stdout:5/40: dwrite d1/f5 [0,4194304] 0 2024-09-20T19:44:44.508 INFO:tasks.workunit.client.1.smithi160.stdout:5/41: chown f0 23480930 1 2024-09-20T19:44:44.509 INFO:tasks.workunit.client.1.smithi160.stdout:5/42: truncate d1/d7/d8/fc 573469 0 2024-09-20T19:44:44.513 INFO:tasks.workunit.client.1.smithi160.stdout:2/47: symlink d7/ld 0 2024-09-20T19:44:44.516 INFO:tasks.workunit.client.1.smithi160.stdout:2/48: creat d7/fe x:0 0 0 2024-09-20T19:44:44.516 INFO:tasks.workunit.client.1.smithi160.stdout:2/49: read - d7/fc zero size 2024-09-20T19:44:44.516 INFO:tasks.workunit.client.1.smithi160.stdout:2/50: write f5 [111223,67634] 0 2024-09-20T19:44:44.516 INFO:tasks.workunit.client.1.smithi160.stdout:2/51: readlink d7/l8 0 2024-09-20T19:44:44.516 INFO:tasks.workunit.client.1.smithi160.stdout:2/52: read f5 [112588,17867] 0 2024-09-20T19:44:44.516 INFO:tasks.workunit.client.1.smithi160.stdout:2/53: chown d7/fa 1 1 2024-09-20T19:44:44.516 INFO:tasks.workunit.client.1.smithi160.stdout:2/54: write d7/fc [685649,109244] 0 2024-09-20T19:44:44.517 INFO:tasks.workunit.client.1.smithi160.stdout:6/42: truncate d8/d9/fd 9833954 0 2024-09-20T19:44:44.517 INFO:tasks.workunit.client.1.smithi160.stdout:2/55: symlink d7/lf 0 2024-09-20T19:44:44.519 INFO:tasks.workunit.client.1.smithi160.stdout:8/33: dwrite d1/f3 [4194304,4194304] 0 2024-09-20T19:44:44.519 INFO:tasks.workunit.client.1.smithi160.stdout:8/34: rename d1 to d1/d7 22 2024-09-20T19:44:44.520 INFO:tasks.workunit.client.1.smithi160.stdout:2/56: mkdir d7/d10 0 2024-09-20T19:44:44.521 INFO:tasks.workunit.client.1.smithi160.stdout:5/43: dread d1/f5 [0,4194304] 0 2024-09-20T19:44:44.522 INFO:tasks.workunit.client.1.smithi160.stdout:8/35: link d1/c5 d1/c8 0 2024-09-20T19:44:44.525 INFO:tasks.workunit.client.1.smithi160.stdout:2/57: rename d7/fc to d7/d10/f11 0 2024-09-20T19:44:44.525 INFO:tasks.workunit.client.1.smithi160.stdout:2/58: write d7/fe [889133,35555] 0 2024-09-20T19:44:44.525 INFO:tasks.workunit.client.1.smithi160.stdout:2/59: stat d7/l8 0 2024-09-20T19:44:44.525 INFO:tasks.workunit.client.1.smithi160.stdout:2/60: write f5 [1111600,115377] 0 2024-09-20T19:44:44.525 INFO:tasks.workunit.client.1.smithi160.stdout:5/44: creat d1/d7/ff x:0 0 0 2024-09-20T19:44:44.528 INFO:tasks.workunit.client.1.smithi160.stdout:8/36: creat d1/f9 x:0 0 0 2024-09-20T19:44:44.530 INFO:tasks.workunit.client.1.smithi160.stdout:2/61: dread d7/d10/f11 [0,4194304] 0 2024-09-20T19:44:44.533 INFO:tasks.workunit.client.1.smithi160.stdout:2/62: symlink d7/l12 0 2024-09-20T19:44:44.538 INFO:tasks.workunit.client.1.smithi160.stdout:3/38: truncate d0/d7/fa 522270 0 2024-09-20T19:44:44.538 INFO:tasks.workunit.client.1.smithi160.stdout:3/39: write d0/f2 [2803650,46152] 0 2024-09-20T19:44:44.540 INFO:tasks.workunit.client.1.smithi160.stdout:3/40: write d0/f2 [1501532,82873] 0 2024-09-20T19:44:44.542 INFO:tasks.workunit.client.1.smithi160.stdout:2/63: dread d7/fe [0,4194304] 0 2024-09-20T19:44:44.542 INFO:tasks.workunit.client.1.smithi160.stdout:4/42: truncate d0/f1 2615887 0 2024-09-20T19:44:44.543 INFO:tasks.workunit.client.1.smithi160.stdout:4/43: chown d0/f1 363408 1 2024-09-20T19:44:44.545 INFO:tasks.workunit.client.1.smithi160.stdout:2/64: mknod d7/c13 0 2024-09-20T19:44:44.550 INFO:tasks.workunit.client.1.smithi160.stdout:2/65: dread f2 [0,4194304] 0 2024-09-20T19:44:44.550 INFO:tasks.workunit.client.1.smithi160.stdout:2/66: write d7/fa [207515,117771] 0 2024-09-20T19:44:44.550 INFO:tasks.workunit.client.1.smithi160.stdout:2/67: truncate d7/fa 1241295 0 2024-09-20T19:44:44.552 INFO:tasks.workunit.client.1.smithi160.stdout:5/45: dwrite d1/d7/d8/fe [0,4194304] 0 2024-09-20T19:44:44.552 INFO:tasks.workunit.client.1.smithi160.stdout:5/46: write d1/d7/ff [887869,16210] 0 2024-09-20T19:44:44.552 INFO:tasks.workunit.client.1.smithi160.stdout:5/47: rename d1 to d1/d7/da/d10 22 2024-09-20T19:44:44.552 INFO:tasks.workunit.client.1.smithi160.stdout:8/37: dwrite f0 [0,4194304] 0 2024-09-20T19:44:44.553 INFO:tasks.workunit.client.1.smithi160.stdout:8/38: dread d1/f4 [0,4194304] 0 2024-09-20T19:44:44.553 INFO:tasks.workunit.client.1.smithi160.stdout:8/39: write d1/f9 [945987,57663] 0 2024-09-20T19:44:44.555 INFO:tasks.workunit.client.1.smithi160.stdout:2/68: dread f5 [0,4194304] 0 2024-09-20T19:44:44.555 INFO:tasks.workunit.client.1.smithi160.stdout:2/69: readlink d7/ld 0 2024-09-20T19:44:44.565 INFO:tasks.workunit.client.1.smithi160.stdout:6/43: dread d8/d9/fd [4194304,4194304] 0 2024-09-20T19:44:44.571 INFO:tasks.workunit.client.1.smithi160.stdout:5/48: symlink d1/d7/l11 0 2024-09-20T19:44:44.573 INFO:tasks.workunit.client.1.smithi160.stdout:8/40: symlink d1/la 0 2024-09-20T19:44:44.576 INFO:tasks.workunit.client.1.smithi160.stdout:2/70: unlink f5 0 2024-09-20T19:44:44.577 INFO:tasks.workunit.client.1.smithi160.stdout:8/41: creat d1/fb x:0 0 0 2024-09-20T19:44:44.582 INFO:tasks.workunit.client.1.smithi160.stdout:6/44: dwrite f5 [0,4194304] 0 2024-09-20T19:44:44.582 INFO:tasks.workunit.client.1.smithi160.stdout:6/45: stat f5 0 2024-09-20T19:44:44.582 INFO:tasks.workunit.client.1.smithi160.stdout:0/38: sync 2024-09-20T19:44:44.582 INFO:tasks.workunit.client.1.smithi160.stdout:0/39: stat d4/d6 0 2024-09-20T19:44:44.582 INFO:tasks.workunit.client.1.smithi160.stdout:0/40: write f3 [2717285,64437] 0 2024-09-20T19:44:44.582 INFO:tasks.workunit.client.1.smithi160.stdout:8/42: write d1/f9 [433779,59872] 0 2024-09-20T19:44:44.582 INFO:tasks.workunit.client.1.smithi160.stdout:8/43: readlink d1/la 0 2024-09-20T19:44:44.584 INFO:tasks.workunit.client.1.smithi160.stdout:6/46: creat d8/d9/f10 x:0 0 0 2024-09-20T19:44:44.585 INFO:tasks.workunit.client.1.smithi160.stdout:3/41: truncate d0/f2 562491 0 2024-09-20T19:44:44.591 INFO:tasks.workunit.client.1.smithi160.stdout:8/44: rmdir d1 39 2024-09-20T19:44:44.594 INFO:tasks.workunit.client.1.smithi160.stdout:8/45: mknod d1/cc 0 2024-09-20T19:44:44.596 INFO:tasks.workunit.client.1.smithi160.stdout:8/46: rmdir d1 39 2024-09-20T19:44:44.598 INFO:tasks.workunit.client.1.smithi160.stdout:8/47: mknod d1/cd 0 2024-09-20T19:44:44.600 INFO:tasks.workunit.client.1.smithi160.stdout:8/48: rename d1/f9 to d1/fe 0 2024-09-20T19:44:44.601 INFO:tasks.workunit.client.1.smithi160.stdout:8/49: mkdir d1/df 0 2024-09-20T19:44:44.603 INFO:tasks.workunit.client.1.smithi160.stdout:8/50: creat d1/df/f10 x:0 0 0 2024-09-20T19:44:44.605 INFO:tasks.workunit.client.1.smithi160.stdout:8/51: mknod d1/c11 0 2024-09-20T19:44:44.607 INFO:tasks.workunit.client.1.smithi160.stdout:2/71: dwrite d7/fa [0,4194304] 0 2024-09-20T19:44:44.607 INFO:tasks.workunit.client.1.smithi160.stdout:2/72: fdatasync d7/d10/f11 0 2024-09-20T19:44:44.609 INFO:tasks.workunit.client.1.smithi160.stdout:6/47: dwrite d8/d9/fa [4194304,4194304] 0 2024-09-20T19:44:44.609 INFO:tasks.workunit.client.1.smithi160.stdout:4/44: dwrite d0/f1 [0,4194304] 0 2024-09-20T19:44:44.611 INFO:tasks.workunit.client.1.smithi160.stdout:2/73: rename d7/c13 to d7/d10/c14 0 2024-09-20T19:44:44.613 INFO:tasks.workunit.client.1.smithi160.stdout:4/45: mknod d0/ca 0 2024-09-20T19:44:44.614 INFO:tasks.workunit.client.1.smithi160.stdout:2/74: dread d7/fe [0,4194304] 0 2024-09-20T19:44:44.616 INFO:tasks.workunit.client.1.smithi160.stdout:4/46: creat d0/d2/d9/fb x:0 0 0 2024-09-20T19:44:44.616 INFO:tasks.workunit.client.1.smithi160.stdout:8/52: dread f0 [0,4194304] 0 2024-09-20T19:44:44.618 INFO:tasks.workunit.client.1.smithi160.stdout:8/53: rename d1/c5 to d1/df/c12 0 2024-09-20T19:44:44.620 INFO:tasks.workunit.client.1.smithi160.stdout:6/48: write f4 [8321677,63647] 0 2024-09-20T19:44:44.620 INFO:tasks.workunit.client.1.smithi160.stdout:8/54: mkdir d1/df/d13 0 2024-09-20T19:44:44.623 INFO:tasks.workunit.client.1.smithi160.stdout:2/75: write d7/fa [3786953,121192] 0 2024-09-20T19:44:44.623 INFO:tasks.workunit.client.1.smithi160.stdout:2/76: chown f2 10206488 1 2024-09-20T19:44:44.623 INFO:tasks.workunit.client.1.smithi160.stdout:6/49: rename f2 to d8/f11 0 2024-09-20T19:44:44.625 INFO:tasks.workunit.client.1.smithi160.stdout:8/55: dread d1/fe [0,4194304] 0 2024-09-20T19:44:44.625 INFO:tasks.workunit.client.1.smithi160.stdout:8/56: chown d1/c8 996511662 1 2024-09-20T19:44:44.625 INFO:tasks.workunit.client.1.smithi160.stdout:4/47: dread d0/f1 [0,4194304] 0 2024-09-20T19:44:44.625 INFO:tasks.workunit.client.1.smithi160.stdout:4/48: dread - d0/d2/d9/fb zero size 2024-09-20T19:44:44.625 INFO:tasks.workunit.client.1.smithi160.stdout:4/49: dread - d0/d2/d9/fb zero size 2024-09-20T19:44:44.626 INFO:tasks.workunit.client.1.smithi160.stdout:9/44: sync 2024-09-20T19:44:44.626 INFO:tasks.workunit.client.1.smithi160.stdout:7/24: sync 2024-09-20T19:44:44.626 INFO:tasks.workunit.client.1.smithi160.stdout:1/17: sync 2024-09-20T19:44:44.626 INFO:tasks.workunit.client.1.smithi160.stdout:7/25: write f1 [1172692,26554] 0 2024-09-20T19:44:44.628 INFO:tasks.workunit.client.1.smithi160.stdout:4/50: creat d0/fc x:0 0 0 2024-09-20T19:44:44.631 INFO:tasks.workunit.client.1.smithi160.stdout:1/18: rename f0 to f3 0 2024-09-20T19:44:44.633 INFO:tasks.workunit.client.1.smithi160.stdout:7/26: rename l2 to d3/d4/lb 0 2024-09-20T19:44:44.633 INFO:tasks.workunit.client.1.smithi160.stdout:7/27: dread - f0 zero size 2024-09-20T19:44:44.634 INFO:tasks.workunit.client.1.smithi160.stdout:6/50: dread d8/f11 [0,4194304] 0 2024-09-20T19:44:44.634 INFO:tasks.workunit.client.1.smithi160.stdout:6/51: readlink d8/lf 0 2024-09-20T19:44:44.634 INFO:tasks.workunit.client.1.smithi160.stdout:6/52: fdatasync d8/d9/fd 0 2024-09-20T19:44:44.634 INFO:tasks.workunit.client.1.smithi160.stdout:6/53: truncate d8/d9/f10 819019 0 2024-09-20T19:44:44.634 INFO:tasks.workunit.client.1.smithi160.stdout:4/51: mkdir d0/d4/dd 0 2024-09-20T19:44:44.637 INFO:tasks.workunit.client.1.smithi160.stdout:9/45: dread d0/fc [0,4194304] 0 2024-09-20T19:44:44.637 INFO:tasks.workunit.client.1.smithi160.stdout:9/46: write d0/fa [890333,67375] 0 2024-09-20T19:44:44.637 INFO:tasks.workunit.client.1.smithi160.stdout:1/19: rename c2 to c4 0 2024-09-20T19:44:44.637 INFO:tasks.workunit.client.1.smithi160.stdout:1/20: rmdir - no directory 2024-09-20T19:44:44.637 INFO:tasks.workunit.client.1.smithi160.stdout:9/47: read d0/fa [70268,99233] 0 2024-09-20T19:44:44.638 INFO:tasks.workunit.client.1.smithi160.stdout:7/28: rename d3/c7 to d3/d4/cc 0 2024-09-20T19:44:44.640 INFO:tasks.workunit.client.1.smithi160.stdout:6/54: rename d8/d9/f10 to d8/d9/f12 0 2024-09-20T19:44:44.642 INFO:tasks.workunit.client.1.smithi160.stdout:4/52: creat d0/d2/d9/fe x:0 0 0 2024-09-20T19:44:44.644 INFO:tasks.workunit.client.1.smithi160.stdout:1/21: creat f5 x:0 0 0 2024-09-20T19:44:44.644 INFO:tasks.workunit.client.1.smithi160.stdout:1/22: truncate f3 75129 0 2024-09-20T19:44:44.644 INFO:tasks.workunit.client.1.smithi160.stdout:1/23: chown f3 101926001 1 2024-09-20T19:44:44.644 INFO:tasks.workunit.client.1.smithi160.stdout:1/24: chown f5 256620 1 2024-09-20T19:44:44.645 INFO:tasks.workunit.client.1.smithi160.stdout:1/25: write f5 [912805,34161] 0 2024-09-20T19:44:44.645 INFO:tasks.workunit.client.1.smithi160.stdout:1/26: read f3 [5327,105061] 0 2024-09-20T19:44:44.645 INFO:tasks.workunit.client.1.smithi160.stdout:9/48: truncate d0/f2 4120643 0 2024-09-20T19:44:44.645 INFO:tasks.workunit.client.1.smithi160.stdout:9/49: write d0/fa [1098027,213] 0 2024-09-20T19:44:44.646 INFO:tasks.workunit.client.1.smithi160.stdout:1/27: dread f3 [0,4194304] 0 2024-09-20T19:44:44.646 INFO:tasks.workunit.client.1.smithi160.stdout:1/28: chown f3 1 1 2024-09-20T19:44:44.649 INFO:tasks.workunit.client.1.smithi160.stdout:6/55: mknod d8/c13 0 2024-09-20T19:44:44.649 INFO:tasks.workunit.client.1.smithi160.stdout:2/77: write d7/fa [5223594,62604] 0 2024-09-20T19:44:44.649 INFO:tasks.workunit.client.1.smithi160.stdout:4/53: symlink d0/d2/d9/lf 0 2024-09-20T19:44:44.651 INFO:tasks.workunit.client.1.smithi160.stdout:2/78: dread d7/fe [0,4194304] 0 2024-09-20T19:44:44.653 INFO:tasks.workunit.client.1.smithi160.stdout:1/29: dread f5 [0,4194304] 0 2024-09-20T19:44:44.653 INFO:tasks.workunit.client.1.smithi160.stdout:9/50: unlink d0/d4/fb 0 2024-09-20T19:44:44.653 INFO:tasks.workunit.client.1.smithi160.stdout:9/51: rename d0 to d0/d8/dd 22 2024-09-20T19:44:44.655 INFO:tasks.workunit.client.1.smithi160.stdout:6/56: symlink d8/l14 0 2024-09-20T19:44:44.655 INFO:tasks.workunit.client.1.smithi160.stdout:6/57: write f4 [10129149,99633] 0 2024-09-20T19:44:44.655 INFO:tasks.workunit.client.1.smithi160.stdout:6/58: readlink d8/lb 0 2024-09-20T19:44:44.656 INFO:tasks.workunit.client.1.smithi160.stdout:4/54: mkdir d0/d10 0 2024-09-20T19:44:44.657 INFO:tasks.workunit.client.1.smithi160.stdout:2/79: creat d7/d10/f15 x:0 0 0 2024-09-20T19:44:44.658 INFO:tasks.workunit.client.1.smithi160.stdout:2/80: read d7/fa [4707679,107082] 0 2024-09-20T19:44:44.658 INFO:tasks.workunit.client.1.smithi160.stdout:2/81: chown d7/l12 79336552 1 2024-09-20T19:44:44.660 INFO:tasks.workunit.client.1.smithi160.stdout:9/52: getdents d0/d8 0 2024-09-20T19:44:44.662 INFO:tasks.workunit.client.1.smithi160.stdout:6/59: unlink d8/d9/fc 0 2024-09-20T19:44:44.663 INFO:tasks.workunit.client.1.smithi160.stdout:4/55: mkdir d0/d11 0 2024-09-20T19:44:44.667 INFO:tasks.workunit.client.1.smithi160.stdout:2/82: creat d7/d10/f16 x:0 0 0 2024-09-20T19:44:44.667 INFO:tasks.workunit.client.1.smithi160.stdout:2/83: write d7/fa [1738923,80504] 0 2024-09-20T19:44:44.667 INFO:tasks.workunit.client.1.smithi160.stdout:2/84: dread - d7/d10/f16 zero size 2024-09-20T19:44:44.668 INFO:tasks.workunit.client.1.smithi160.stdout:9/53: mkdir d0/de 0 2024-09-20T19:44:44.669 INFO:tasks.workunit.client.1.smithi160.stdout:6/60: unlink f4 0 2024-09-20T19:44:44.670 INFO:tasks.workunit.client.1.smithi160.stdout:6/61: chown c6 362285 1 2024-09-20T19:44:44.670 INFO:tasks.workunit.client.1.smithi160.stdout:6/62: truncate d8/d9/fe 844392 0 2024-09-20T19:44:44.670 INFO:tasks.workunit.client.1.smithi160.stdout:4/56: rmdir d0/d4 39 2024-09-20T19:44:44.671 INFO:tasks.workunit.client.1.smithi160.stdout:6/63: fsync d8/f11 0 2024-09-20T19:44:44.672 INFO:tasks.workunit.client.1.smithi160.stdout:2/85: creat d7/f17 x:0 0 0 2024-09-20T19:44:44.676 INFO:tasks.workunit.client.1.smithi160.stdout:4/57: mknod d0/d2/c12 0 2024-09-20T19:44:44.680 INFO:tasks.workunit.client.1.smithi160.stdout:6/64: creat d8/d9/f15 x:0 0 0 2024-09-20T19:44:44.680 INFO:tasks.workunit.client.1.smithi160.stdout:6/65: chown d8 11538432 1 2024-09-20T19:44:44.680 INFO:tasks.workunit.client.1.smithi160.stdout:6/66: chown d8/d9/fa 432206 1 2024-09-20T19:44:44.681 INFO:tasks.workunit.client.1.smithi160.stdout:4/58: dread d0/f1 [0,4194304] 0 2024-09-20T19:44:44.682 INFO:tasks.workunit.client.1.smithi160.stdout:2/86: symlink d7/l18 0 2024-09-20T19:44:44.686 INFO:tasks.workunit.client.1.smithi160.stdout:1/30: getdents . 0 2024-09-20T19:44:44.686 INFO:tasks.workunit.client.1.smithi160.stdout:1/31: rmdir - no directory 2024-09-20T19:44:44.687 INFO:tasks.workunit.client.1.smithi160.stdout:4/59: write d0/f1 [2828823,112341] 0 2024-09-20T19:44:44.694 INFO:tasks.workunit.client.1.smithi160.stdout:2/87: symlink d7/l19 0 2024-09-20T19:44:44.694 INFO:tasks.workunit.client.1.smithi160.stdout:2/88: read d7/fe [10072,6526] 0 2024-09-20T19:44:44.698 INFO:tasks.workunit.client.1.smithi160.stdout:1/32: mkdir d6 0 2024-09-20T19:44:44.700 INFO:tasks.workunit.client.1.smithi160.stdout:9/54: unlink d0/f2 0 2024-09-20T19:44:44.703 INFO:tasks.workunit.client.1.smithi160.stdout:2/89: dread f2 [0,4194304] 0 2024-09-20T19:44:44.704 INFO:tasks.workunit.client.1.smithi160.stdout:2/90: write d7/d10/f11 [675327,23852] 0 2024-09-20T19:44:44.705 INFO:tasks.workunit.client.1.smithi160.stdout:9/55: mknod d0/cf 0 2024-09-20T19:44:44.710 INFO:tasks.workunit.client.1.smithi160.stdout:2/91: link d7/d10/c14 d7/c1a 0 2024-09-20T19:44:44.710 INFO:tasks.workunit.client.1.smithi160.stdout:2/92: write d7/d10/f16 [454351,94738] 0 2024-09-20T19:44:44.711 INFO:tasks.workunit.client.1.smithi160.stdout:2/93: read d7/d10/f16 [65593,58131] 0 2024-09-20T19:44:44.712 INFO:tasks.workunit.client.1.smithi160.stdout:4/60: dwrite d0/f1 [0,4194304] 0 2024-09-20T19:44:44.714 INFO:tasks.workunit.client.1.smithi160.stdout:2/94: rename d7/l8 to d7/l1b 0 2024-09-20T19:44:44.714 INFO:tasks.workunit.client.1.smithi160.stdout:2/95: dread - d7/f17 zero size 2024-09-20T19:44:44.714 INFO:tasks.workunit.client.1.smithi160.stdout:9/56: read d0/fc [2422812,97848] 0 2024-09-20T19:44:44.715 INFO:tasks.workunit.client.1.smithi160.stdout:9/57: read d0/fc [3843082,52035] 0 2024-09-20T19:44:44.716 INFO:tasks.workunit.client.1.smithi160.stdout:2/96: getdents d7/d10 0 2024-09-20T19:44:44.716 INFO:tasks.workunit.client.1.smithi160.stdout:2/97: fsync d7/d10/f15 0 2024-09-20T19:44:44.719 INFO:tasks.workunit.client.1.smithi160.stdout:2/98: creat d7/d10/f1c x:0 0 0 2024-09-20T19:44:44.722 INFO:tasks.workunit.client.1.smithi160.stdout:6/67: truncate d8/d9/fa 8038778 0 2024-09-20T19:44:44.724 INFO:tasks.workunit.client.1.smithi160.stdout:4/61: read d0/f1 [1542883,85732] 0 2024-09-20T19:44:44.724 INFO:tasks.workunit.client.1.smithi160.stdout:4/62: truncate d0/d2/d9/fe 707784 0 2024-09-20T19:44:44.725 INFO:tasks.workunit.client.1.smithi160.stdout:1/33: dwrite f3 [0,4194304] 0 2024-09-20T19:44:44.725 INFO:tasks.workunit.client.1.smithi160.stdout:1/34: write f5 [1018551,17257] 0 2024-09-20T19:44:44.725 INFO:tasks.workunit.client.1.smithi160.stdout:2/99: dread f2 [0,4194304] 0 2024-09-20T19:44:44.726 INFO:tasks.workunit.client.1.smithi160.stdout:4/63: getdents d0 0 2024-09-20T19:44:44.731 INFO:tasks.workunit.client.1.smithi160.stdout:4/64: dread d0/f1 [0,4194304] 0 2024-09-20T19:44:44.731 INFO:tasks.workunit.client.1.smithi160.stdout:4/65: write d0/fc [148998,8131] 0 2024-09-20T19:44:44.733 INFO:tasks.workunit.client.1.smithi160.stdout:4/66: fdatasync d0/fc 0 2024-09-20T19:44:44.733 INFO:tasks.workunit.client.1.smithi160.stdout:4/67: fsync d0/fc 0 2024-09-20T19:44:44.733 INFO:tasks.workunit.client.1.smithi160.stdout:4/68: read d0/fc [147821,31358] 0 2024-09-20T19:44:44.733 INFO:tasks.workunit.client.1.smithi160.stdout:4/69: truncate d0/fc 1035054 0 2024-09-20T19:44:44.733 INFO:tasks.workunit.client.1.smithi160.stdout:4/70: truncate d0/d2/d9/fb 411537 0 2024-09-20T19:44:44.738 INFO:tasks.workunit.client.1.smithi160.stdout:1/35: dread f3 [0,4194304] 0 2024-09-20T19:44:44.739 INFO:tasks.workunit.client.1.smithi160.stdout:5/49: sync 2024-09-20T19:44:44.740 INFO:tasks.workunit.client.1.smithi160.stdout:1/36: link l1 d6/l7 0 2024-09-20T19:44:44.742 INFO:tasks.workunit.client.1.smithi160.stdout:5/50: creat d1/d7/f12 x:0 0 0 2024-09-20T19:44:44.742 INFO:tasks.workunit.client.1.smithi160.stdout:5/51: chown d1/d7/fb 7336 1 2024-09-20T19:44:44.745 INFO:tasks.workunit.client.1.smithi160.stdout:1/37: creat d6/f8 x:0 0 0 2024-09-20T19:44:44.745 INFO:tasks.workunit.client.1.smithi160.stdout:1/38: write d6/f8 [695200,49116] 0 2024-09-20T19:44:44.745 INFO:tasks.workunit.client.1.smithi160.stdout:1/39: write d6/f8 [1397548,74884] 0 2024-09-20T19:44:44.745 INFO:tasks.workunit.client.1.smithi160.stdout:5/52: mkdir d1/d7/d13 0 2024-09-20T19:44:44.746 INFO:tasks.workunit.client.1.smithi160.stdout:4/71: dwrite d0/d2/d9/fb [0,4194304] 0 2024-09-20T19:44:44.751 INFO:tasks.workunit.client.1.smithi160.stdout:2/100: getdents d7 0 2024-09-20T19:44:44.751 INFO:tasks.workunit.client.1.smithi160.stdout:2/101: write d7/f17 [1005029,17931] 0 2024-09-20T19:44:44.754 INFO:tasks.workunit.client.1.smithi160.stdout:4/72: dread d0/f1 [0,4194304] 0 2024-09-20T19:44:44.754 INFO:tasks.workunit.client.1.smithi160.stdout:5/53: dread d1/d7/ff [0,4194304] 0 2024-09-20T19:44:44.762 INFO:tasks.workunit.client.1.smithi160.stdout:4/73: dread d0/f1 [0,4194304] 0 2024-09-20T19:44:44.765 INFO:tasks.workunit.client.1.smithi160.stdout:6/68: dread d8/d9/fd [0,4194304] 0 2024-09-20T19:44:44.766 INFO:tasks.workunit.client.1.smithi160.stdout:4/74: rename d0/d2/c6 to d0/d4/dd/c13 0 2024-09-20T19:44:44.767 INFO:tasks.workunit.client.1.smithi160.stdout:1/40: dwrite f5 [0,4194304] 0 2024-09-20T19:44:44.768 INFO:tasks.workunit.client.1.smithi160.stdout:4/75: dread d0/d2/d9/fe [0,4194304] 0 2024-09-20T19:44:44.768 INFO:tasks.workunit.client.1.smithi160.stdout:4/76: write d0/f1 [3410428,37514] 0 2024-09-20T19:44:44.768 INFO:tasks.workunit.client.1.smithi160.stdout:4/77: getdents d0 0 2024-09-20T19:44:44.770 INFO:tasks.workunit.client.1.smithi160.stdout:6/69: symlink d8/d9/l16 0 2024-09-20T19:44:44.772 INFO:tasks.workunit.client.1.smithi160.stdout:4/78: rmdir d0/d2/d9 39 2024-09-20T19:44:44.775 INFO:tasks.workunit.client.1.smithi160.stdout:1/41: write f5 [654316,113280] 0 2024-09-20T19:44:44.776 INFO:tasks.workunit.client.1.smithi160.stdout:6/70: dread f1 [0,4194304] 0 2024-09-20T19:44:44.776 INFO:tasks.workunit.client.1.smithi160.stdout:6/71: write d8/d9/f15 [896555,13781] 0 2024-09-20T19:44:44.780 INFO:tasks.workunit.client.1.smithi160.stdout:1/42: mknod d6/c9 0 2024-09-20T19:44:44.797 INFO:tasks.workunit.client.1.smithi160.stdout:2/102: dwrite d7/d10/f15 [0,4194304] 0 2024-09-20T19:44:44.797 INFO:tasks.workunit.client.1.smithi160.stdout:5/54: dwrite d1/d7/f9 [0,4194304] 0 2024-09-20T19:44:44.797 INFO:tasks.workunit.client.1.smithi160.stdout:5/55: stat d1/f5 0 2024-09-20T19:44:44.798 INFO:tasks.workunit.client.1.smithi160.stdout:4/79: dwrite d0/d2/d9/fb [0,4194304] 0 2024-09-20T19:44:44.800 INFO:tasks.workunit.client.1.smithi160.stdout:4/80: dread d0/d2/d9/fe [0,4194304] 0 2024-09-20T19:44:44.802 INFO:tasks.workunit.client.1.smithi160.stdout:2/103: unlink f2 0 2024-09-20T19:44:44.811 INFO:tasks.workunit.client.1.smithi160.stdout:1/43: dwrite f5 [0,4194304] 0 2024-09-20T19:44:44.816 INFO:tasks.workunit.client.1.smithi160.stdout:4/81: mkdir d0/d10/d14 0 2024-09-20T19:44:44.817 INFO:tasks.workunit.client.1.smithi160.stdout:2/104: unlink d7/l18 0 2024-09-20T19:44:44.821 INFO:tasks.workunit.client.1.smithi160.stdout:4/82: creat d0/d4/dd/f15 x:0 0 0 2024-09-20T19:44:44.822 INFO:tasks.workunit.client.1.smithi160.stdout:4/83: write d0/f1 [2634801,52399] 0 2024-09-20T19:44:44.824 INFO:tasks.workunit.client.1.smithi160.stdout:4/84: mknod d0/d2/d9/c16 0 2024-09-20T19:44:44.826 INFO:tasks.workunit.client.1.smithi160.stdout:2/105: rename d7/d10/f11 to d7/f1d 0 2024-09-20T19:44:44.826 INFO:tasks.workunit.client.1.smithi160.stdout:2/106: write d7/fa [451384,103730] 0 2024-09-20T19:44:44.828 INFO:tasks.workunit.client.1.smithi160.stdout:4/85: rename d0/fc to d0/d2/d9/f17 0 2024-09-20T19:44:44.830 INFO:tasks.workunit.client.1.smithi160.stdout:2/107: creat d7/d10/f1e x:0 0 0 2024-09-20T19:44:44.831 INFO:tasks.workunit.client.1.smithi160.stdout:2/108: dread - d7/d10/f1e zero size 2024-09-20T19:44:44.833 INFO:tasks.workunit.client.1.smithi160.stdout:2/109: dread d7/f17 [0,4194304] 0 2024-09-20T19:44:44.834 INFO:tasks.workunit.client.1.smithi160.stdout:2/110: creat d7/f1f x:0 0 0 2024-09-20T19:44:44.838 INFO:tasks.workunit.client.1.smithi160.stdout:1/44: dwrite f3 [0,4194304] 0 2024-09-20T19:44:44.840 INFO:tasks.workunit.client.1.smithi160.stdout:4/86: dread d0/d2/d9/fb [0,4194304] 0 2024-09-20T19:44:44.842 INFO:tasks.workunit.client.1.smithi160.stdout:1/45: mkdir d6/da 0 2024-09-20T19:44:44.844 INFO:tasks.workunit.client.1.smithi160.stdout:4/87: creat d0/d10/f18 x:0 0 0 2024-09-20T19:44:44.845 INFO:tasks.workunit.client.1.smithi160.stdout:4/88: mkdir d0/d19 0 2024-09-20T19:44:44.847 INFO:tasks.workunit.client.1.smithi160.stdout:4/89: symlink d0/d2/d9/l1a 0 2024-09-20T19:44:44.847 INFO:tasks.workunit.client.1.smithi160.stdout:4/90: write d0/d10/f18 [90024,130693] 0 2024-09-20T19:44:44.848 INFO:tasks.workunit.client.1.smithi160.stdout:1/46: read f5 [4160232,116625] 0 2024-09-20T19:44:44.849 INFO:tasks.workunit.client.1.smithi160.stdout:1/47: chown d6/da 5 1 2024-09-20T19:44:44.852 INFO:tasks.workunit.client.1.smithi160.stdout:1/48: dread d6/f8 [0,4194304] 0 2024-09-20T19:44:44.856 INFO:tasks.workunit.client.1.smithi160.stdout:1/49: dread f5 [0,4194304] 0 2024-09-20T19:44:44.858 INFO:tasks.workunit.client.1.smithi160.stdout:8/57: sync 2024-09-20T19:44:44.858 INFO:tasks.workunit.client.1.smithi160.stdout:3/42: sync 2024-09-20T19:44:44.858 INFO:tasks.workunit.client.1.smithi160.stdout:0/41: sync 2024-09-20T19:44:44.859 INFO:tasks.workunit.client.1.smithi160.stdout:7/29: sync 2024-09-20T19:44:44.859 INFO:tasks.workunit.client.1.smithi160.stdout:3/43: write d0/f2 [18490,88314] 0 2024-09-20T19:44:44.863 INFO:tasks.workunit.client.1.smithi160.stdout:8/58: truncate d1/f3 2758876 0 2024-09-20T19:44:44.863 INFO:tasks.workunit.client.1.smithi160.stdout:8/59: stat d1/fb 0 2024-09-20T19:44:44.863 INFO:tasks.workunit.client.1.smithi160.stdout:0/42: mknod d4/d6/cc 0 2024-09-20T19:44:44.866 INFO:tasks.workunit.client.1.smithi160.stdout:8/60: unlink d1/df/f10 0 2024-09-20T19:44:44.868 INFO:tasks.workunit.client.1.smithi160.stdout:4/91: dwrite d0/d2/d9/fb [0,4194304] 0 2024-09-20T19:44:44.869 INFO:tasks.workunit.client.1.smithi160.stdout:0/43: dread d4/f8 [0,4194304] 0 2024-09-20T19:44:44.869 INFO:tasks.workunit.client.1.smithi160.stdout:0/44: readlink - no filename 2024-09-20T19:44:44.877 INFO:tasks.workunit.client.1.smithi160.stdout:4/92: link d0/d4/dd/c13 d0/d10/c1b 0 2024-09-20T19:44:44.880 INFO:tasks.workunit.client.1.smithi160.stdout:4/93: rename d0/d2/d9/fe to d0/d2/d9/f1c 0 2024-09-20T19:44:44.880 INFO:tasks.workunit.client.1.smithi160.stdout:4/94: chown d0 72089141 1 2024-09-20T19:44:44.881 INFO:tasks.workunit.client.1.smithi160.stdout:2/111: truncate d7/fa 1271105 0 2024-09-20T19:44:44.885 INFO:tasks.workunit.client.1.smithi160.stdout:4/95: rename d0/d4/dd/c13 to d0/d11/c1d 0 2024-09-20T19:44:44.887 INFO:tasks.workunit.client.1.smithi160.stdout:2/112: creat d7/d10/f20 x:0 0 0 2024-09-20T19:44:44.888 INFO:tasks.workunit.client.1.smithi160.stdout:3/44: dwrite d0/d7/fa [0,4194304] 0 2024-09-20T19:44:44.890 INFO:tasks.workunit.client.1.smithi160.stdout:4/96: symlink d0/d2/l1e 0 2024-09-20T19:44:44.891 INFO:tasks.workunit.client.1.smithi160.stdout:4/97: dread d0/d2/d9/f1c [0,4194304] 0 2024-09-20T19:44:44.894 INFO:tasks.workunit.client.1.smithi160.stdout:3/45: rmdir d0 39 2024-09-20T19:44:44.894 INFO:tasks.workunit.client.1.smithi160.stdout:1/50: dwrite f5 [0,4194304] 0 2024-09-20T19:44:44.894 INFO:tasks.workunit.client.1.smithi160.stdout:1/51: fsync f5 0 2024-09-20T19:44:44.895 INFO:tasks.workunit.client.1.smithi160.stdout:4/98: mknod d0/d19/c1f 0 2024-09-20T19:44:44.897 INFO:tasks.workunit.client.1.smithi160.stdout:3/46: link d0/c5 d0/d6/ce 0 2024-09-20T19:44:44.900 INFO:tasks.workunit.client.1.smithi160.stdout:1/52: creat d6/da/fb x:0 0 0 2024-09-20T19:44:44.902 INFO:tasks.workunit.client.1.smithi160.stdout:3/47: creat d0/ff x:0 0 0 2024-09-20T19:44:44.902 INFO:tasks.workunit.client.1.smithi160.stdout:3/48: readlink d0/ld 0 2024-09-20T19:44:44.902 INFO:tasks.workunit.client.1.smithi160.stdout:3/49: write d0/f2 [291600,12390] 0 2024-09-20T19:44:44.902 INFO:tasks.workunit.client.1.smithi160.stdout:3/50: fsync d0/d7/fa 0 2024-09-20T19:44:44.903 INFO:tasks.workunit.client.1.smithi160.stdout:8/61: dwrite d1/f4 [0,4194304] 0 2024-09-20T19:44:44.904 INFO:tasks.workunit.client.1.smithi160.stdout:0/45: dwrite f2 [0,4194304] 0 2024-09-20T19:44:44.904 INFO:tasks.workunit.client.1.smithi160.stdout:0/46: read - d4/d6/fa zero size 2024-09-20T19:44:44.905 INFO:tasks.workunit.client.1.smithi160.stdout:3/51: mknod d0/d7/c10 0 2024-09-20T19:44:44.905 INFO:tasks.workunit.client.1.smithi160.stdout:3/52: chown d0/ff 687 1 2024-09-20T19:44:44.906 INFO:tasks.workunit.client.1.smithi160.stdout:3/53: chown d0/d7 6834 1 2024-09-20T19:44:44.906 INFO:tasks.workunit.client.1.smithi160.stdout:3/54: truncate d0/ff 50927 0 2024-09-20T19:44:44.908 INFO:tasks.workunit.client.1.smithi160.stdout:7/30: dwrite f0 [0,4194304] 0 2024-09-20T19:44:44.910 INFO:tasks.workunit.client.1.smithi160.stdout:8/62: mkdir d1/d14 0 2024-09-20T19:44:44.913 INFO:tasks.workunit.client.1.smithi160.stdout:7/31: dread f1 [0,4194304] 0 2024-09-20T19:44:44.914 INFO:tasks.workunit.client.1.smithi160.stdout:0/47: mknod d4/d6/db/cd 0 2024-09-20T19:44:44.917 INFO:tasks.workunit.client.1.smithi160.stdout:3/55: creat d0/d7/f11 x:0 0 0 2024-09-20T19:44:44.920 INFO:tasks.workunit.client.1.smithi160.stdout:4/99: dwrite d0/d2/d9/fb [4194304,4194304] 0 2024-09-20T19:44:44.920 INFO:tasks.workunit.client.1.smithi160.stdout:8/63: read d1/f4 [1282844,120472] 0 2024-09-20T19:44:44.920 INFO:tasks.workunit.client.1.smithi160.stdout:8/64: fdatasync d1/f3 0 2024-09-20T19:44:44.927 INFO:tasks.workunit.client.1.smithi160.stdout:7/32: truncate d3/d4/f8 1794532 0 2024-09-20T19:44:44.929 INFO:tasks.workunit.client.1.smithi160.stdout:3/56: creat d0/db/f12 x:0 0 0 2024-09-20T19:44:44.929 INFO:tasks.workunit.client.1.smithi160.stdout:3/57: fdatasync d0/d7/f11 0 2024-09-20T19:44:44.929 INFO:tasks.workunit.client.1.smithi160.stdout:9/58: sync 2024-09-20T19:44:44.929 INFO:tasks.workunit.client.1.smithi160.stdout:9/59: chown d0/d4/d5 1 1 2024-09-20T19:44:44.929 INFO:tasks.workunit.client.1.smithi160.stdout:9/60: truncate d0/fa 1455739 0 2024-09-20T19:44:44.930 INFO:tasks.workunit.client.1.smithi160.stdout:4/100: rmdir d0/d10 39 2024-09-20T19:44:44.933 INFO:tasks.workunit.client.1.smithi160.stdout:8/65: creat d1/df/d13/f15 x:0 0 0 2024-09-20T19:44:44.935 INFO:tasks.workunit.client.1.smithi160.stdout:7/33: rmdir d3/d9 39 2024-09-20T19:44:44.935 INFO:tasks.workunit.client.1.smithi160.stdout:7/34: chown d3/d4 953 1 2024-09-20T19:44:44.937 INFO:tasks.workunit.client.1.smithi160.stdout:3/58: link d0/c1 d0/c13 0 2024-09-20T19:44:44.937 INFO:tasks.workunit.client.1.smithi160.stdout:9/61: symlink d0/l10 0 2024-09-20T19:44:44.943 INFO:tasks.workunit.client.1.smithi160.stdout:9/62: dread d0/fa [0,4194304] 0 2024-09-20T19:44:44.943 INFO:tasks.workunit.client.1.smithi160.stdout:9/63: readlink d0/l3 0 2024-09-20T19:44:44.948 INFO:tasks.workunit.client.1.smithi160.stdout:1/53: getdents d6/da 0 2024-09-20T19:44:44.949 INFO:tasks.workunit.client.1.smithi160.stdout:7/35: write f0 [401603,19821] 0 2024-09-20T19:44:44.950 INFO:tasks.workunit.client.1.smithi160.stdout:1/54: rename d6/c9 to d6/da/cc 0 2024-09-20T19:44:44.950 INFO:tasks.workunit.client.1.smithi160.stdout:3/59: write d0/d7/fa [1059058,128553] 0 2024-09-20T19:44:44.951 INFO:tasks.workunit.client.1.smithi160.stdout:9/64: dread d0/fa [0,4194304] 0 2024-09-20T19:44:44.952 INFO:tasks.workunit.client.1.smithi160.stdout:7/36: mknod d3/d9/cd 0 2024-09-20T19:44:44.954 INFO:tasks.workunit.client.1.smithi160.stdout:2/113: dwrite d7/fa [0,4194304] 0 2024-09-20T19:44:44.954 INFO:tasks.workunit.client.1.smithi160.stdout:7/37: write f1 [682293,45798] 0 2024-09-20T19:44:44.954 INFO:tasks.workunit.client.1.smithi160.stdout:8/66: dwrite d1/f4 [0,4194304] 0 2024-09-20T19:44:44.954 INFO:tasks.workunit.client.1.smithi160.stdout:8/67: chown f0 0 1 2024-09-20T19:44:44.957 INFO:tasks.workunit.client.1.smithi160.stdout:3/60: symlink d0/d7/d9/l14 0 2024-09-20T19:44:44.959 INFO:tasks.workunit.client.1.smithi160.stdout:9/65: link d0/d8/l9 d0/d4/d5/l11 0 2024-09-20T19:44:44.960 INFO:tasks.workunit.client.1.smithi160.stdout:4/101: dwrite d0/d2/d9/f17 [0,4194304] 0 2024-09-20T19:44:44.965 INFO:tasks.workunit.client.1.smithi160.stdout:8/68: write d1/f3 [928461,88031] 0 2024-09-20T19:44:44.965 INFO:tasks.workunit.client.1.smithi160.stdout:8/69: dread - d1/df/d13/f15 zero size 2024-09-20T19:44:44.965 INFO:tasks.workunit.client.1.smithi160.stdout:1/55: mknod d6/da/cd 0 2024-09-20T19:44:44.966 INFO:tasks.workunit.client.1.smithi160.stdout:9/66: mknod d0/c12 0 2024-09-20T19:44:44.966 INFO:tasks.workunit.client.1.smithi160.stdout:9/67: write d0/fa [438989,43134] 0 2024-09-20T19:44:44.966 INFO:tasks.workunit.client.1.smithi160.stdout:9/68: write d0/fa [1691044,68403] 0 2024-09-20T19:44:44.966 INFO:tasks.workunit.client.1.smithi160.stdout:5/56: sync 2024-09-20T19:44:44.966 INFO:tasks.workunit.client.1.smithi160.stdout:6/72: sync 2024-09-20T19:44:44.966 INFO:tasks.workunit.client.1.smithi160.stdout:5/57: stat d1/d7/d8/fc 0 2024-09-20T19:44:44.966 INFO:tasks.workunit.client.1.smithi160.stdout:6/73: write d8/d9/f12 [1012020,66660] 0 2024-09-20T19:44:44.967 INFO:tasks.workunit.client.1.smithi160.stdout:2/114: symlink d7/d10/l21 0 2024-09-20T19:44:44.971 INFO:tasks.workunit.client.1.smithi160.stdout:4/102: creat d0/d19/f20 x:0 0 0 2024-09-20T19:44:44.972 INFO:tasks.workunit.client.1.smithi160.stdout:3/61: dwrite d0/d7/f11 [0,4194304] 0 2024-09-20T19:44:44.977 INFO:tasks.workunit.client.1.smithi160.stdout:9/69: rmdir d0/d4/d5 39 2024-09-20T19:44:44.978 INFO:tasks.workunit.client.1.smithi160.stdout:5/58: symlink d1/d7/l14 0 2024-09-20T19:44:44.978 INFO:tasks.workunit.client.1.smithi160.stdout:8/70: dread d1/f3 [0,4194304] 0 2024-09-20T19:44:44.979 INFO:tasks.workunit.client.1.smithi160.stdout:6/74: symlink d8/l17 0 2024-09-20T19:44:44.980 INFO:tasks.workunit.client.1.smithi160.stdout:4/103: dread d0/f1 [0,4194304] 0 2024-09-20T19:44:44.980 INFO:tasks.workunit.client.1.smithi160.stdout:2/115: chown d7/d10/c14 845839334 1 2024-09-20T19:44:44.987 INFO:tasks.workunit.client.1.smithi160.stdout:9/70: unlink d0/d4/d5/l11 0 2024-09-20T19:44:44.987 INFO:tasks.workunit.client.1.smithi160.stdout:5/59: truncate d1/d7/f9 1066753 0 2024-09-20T19:44:44.990 INFO:tasks.workunit.client.1.smithi160.stdout:8/71: creat d1/df/d13/f16 x:0 0 0 2024-09-20T19:44:44.990 INFO:tasks.workunit.client.1.smithi160.stdout:8/72: dread - d1/df/d13/f15 zero size 2024-09-20T19:44:44.990 INFO:tasks.workunit.client.1.smithi160.stdout:8/73: truncate d1/df/d13/f16 812184 0 2024-09-20T19:44:44.991 INFO:tasks.workunit.client.1.smithi160.stdout:6/75: mknod d8/c18 0 2024-09-20T19:44:44.994 INFO:tasks.workunit.client.1.smithi160.stdout:9/71: truncate d0/fc 2211211 0 2024-09-20T19:44:44.995 INFO:tasks.workunit.client.1.smithi160.stdout:6/76: dread f1 [0,4194304] 0 2024-09-20T19:44:44.995 INFO:tasks.workunit.client.1.smithi160.stdout:5/60: mknod d1/d7/d8/c15 0 2024-09-20T19:44:44.996 INFO:tasks.workunit.client.1.smithi160.stdout:8/74: creat d1/f17 x:0 0 0 2024-09-20T19:44:44.997 INFO:tasks.workunit.client.1.smithi160.stdout:3/62: dwrite d0/ff [0,4194304] 0 2024-09-20T19:44:44.997 INFO:tasks.workunit.client.1.smithi160.stdout:4/104: dwrite d0/d4/dd/f15 [0,4194304] 0 2024-09-20T19:44:44.997 INFO:tasks.workunit.client.1.smithi160.stdout:4/105: chown d0/d2/d9/c16 87507 1 2024-09-20T19:44:44.998 INFO:tasks.workunit.client.1.smithi160.stdout:2/116: mkdir d7/d10/d22 0 2024-09-20T19:44:44.999 INFO:tasks.workunit.client.1.smithi160.stdout:9/72: mkdir d0/d4/d13 0 2024-09-20T19:44:45.003 INFO:tasks.workunit.client.1.smithi160.stdout:6/77: truncate d8/d9/fa 4024746 0 2024-09-20T19:44:45.004 INFO:tasks.workunit.client.1.smithi160.stdout:5/61: creat d1/d7/da/f16 x:0 0 0 2024-09-20T19:44:45.004 INFO:tasks.workunit.client.1.smithi160.stdout:5/62: readlink d1/d7/l11 0 2024-09-20T19:44:45.012 INFO:tasks.workunit.client.1.smithi160.stdout:4/106: unlink d0/ca 0 2024-09-20T19:44:45.012 INFO:tasks.workunit.client.1.smithi160.stdout:8/75: dwrite d1/f4 [4194304,4194304] 0 2024-09-20T19:44:45.013 INFO:tasks.workunit.client.1.smithi160.stdout:3/63: creat d0/db/dc/f15 x:0 0 0 2024-09-20T19:44:45.013 INFO:tasks.workunit.client.1.smithi160.stdout:3/64: write d0/d7/fa [1375644,97792] 0 2024-09-20T19:44:45.019 INFO:tasks.workunit.client.1.smithi160.stdout:2/117: symlink d7/l23 0 2024-09-20T19:44:45.020 INFO:tasks.workunit.client.1.smithi160.stdout:9/73: creat d0/f14 x:0 0 0 2024-09-20T19:44:45.022 INFO:tasks.workunit.client.1.smithi160.stdout:4/107: symlink d0/d19/l21 0 2024-09-20T19:44:45.022 INFO:tasks.workunit.client.1.smithi160.stdout:4/108: chown d0/d4/dd/f15 115532973 1 2024-09-20T19:44:45.023 INFO:tasks.workunit.client.1.smithi160.stdout:5/63: dwrite d1/d7/da/f16 [0,4194304] 0 2024-09-20T19:44:45.025 INFO:tasks.workunit.client.1.smithi160.stdout:3/65: rmdir d0/d7 39 2024-09-20T19:44:45.027 INFO:tasks.workunit.client.1.smithi160.stdout:2/118: mkdir d7/d10/d22/d24 0 2024-09-20T19:44:45.029 INFO:tasks.workunit.client.1.smithi160.stdout:9/74: link d0/l3 d0/d4/d13/l15 0 2024-09-20T19:44:45.029 INFO:tasks.workunit.client.1.smithi160.stdout:9/75: readlink d0/d4/d13/l15 0 2024-09-20T19:44:45.029 INFO:tasks.workunit.client.1.smithi160.stdout:9/76: readlink d0/d4/d13/l15 0 2024-09-20T19:44:45.029 INFO:tasks.workunit.client.1.smithi160.stdout:9/77: stat d0/d4 0 2024-09-20T19:44:45.031 INFO:tasks.workunit.client.1.smithi160.stdout:4/109: mkdir d0/d11/d22 0 2024-09-20T19:44:45.036 INFO:tasks.workunit.client.1.smithi160.stdout:5/64: mkdir d1/d7/da/d17 0 2024-09-20T19:44:45.037 INFO:tasks.workunit.client.1.smithi160.stdout:5/65: dread d1/d7/d8/fc [0,4194304] 0 2024-09-20T19:44:45.041 INFO:tasks.workunit.client.1.smithi160.stdout:3/66: mknod d0/db/c16 0 2024-09-20T19:44:45.041 INFO:tasks.workunit.client.1.smithi160.stdout:3/67: stat d0 0 2024-09-20T19:44:45.041 INFO:tasks.workunit.client.1.smithi160.stdout:3/68: truncate d0/db/dc/f15 441492 0 2024-09-20T19:44:45.042 INFO:tasks.workunit.client.1.smithi160.stdout:2/119: creat d7/f25 x:0 0 0 2024-09-20T19:44:45.045 INFO:tasks.workunit.client.1.smithi160.stdout:3/69: mknod d0/d7/d9/c17 0 2024-09-20T19:44:45.045 INFO:tasks.workunit.client.1.smithi160.stdout:3/70: chown d0/db/c16 15049352 1 2024-09-20T19:44:45.046 INFO:tasks.workunit.client.1.smithi160.stdout:5/66: dread f0 [0,4194304] 0 2024-09-20T19:44:45.046 INFO:tasks.workunit.client.1.smithi160.stdout:5/67: chown d1/d7/l14 2895 1 2024-09-20T19:44:45.047 INFO:tasks.workunit.client.1.smithi160.stdout:2/120: write d7/d10/f16 [518058,26269] 0 2024-09-20T19:44:45.050 INFO:tasks.workunit.client.1.smithi160.stdout:3/71: rmdir d0/d7 39 2024-09-20T19:44:45.050 INFO:tasks.workunit.client.1.smithi160.stdout:3/72: readlink d0/d7/d9/l14 0 2024-09-20T19:44:45.053 INFO:tasks.workunit.client.1.smithi160.stdout:5/68: rename d1/d7/fd to d1/d7/d8/f18 0 2024-09-20T19:44:45.053 INFO:tasks.workunit.client.1.smithi160.stdout:5/69: getdents d1/d7/da/d17 0 2024-09-20T19:44:45.053 INFO:tasks.workunit.client.1.smithi160.stdout:5/70: fsync d1/f5 0 2024-09-20T19:44:45.053 INFO:tasks.workunit.client.1.smithi160.stdout:3/73: symlink d0/d7/d9/l18 0 2024-09-20T19:44:45.055 INFO:tasks.workunit.client.1.smithi160.stdout:9/78: dwrite d0/fa [0,4194304] 0 2024-09-20T19:44:45.056 INFO:tasks.workunit.client.1.smithi160.stdout:9/79: write d0/fc [1805020,116374] 0 2024-09-20T19:44:45.056 INFO:tasks.workunit.client.1.smithi160.stdout:9/80: read - d0/f14 zero size 2024-09-20T19:44:45.056 INFO:tasks.workunit.client.1.smithi160.stdout:9/81: fdatasync d0/fa 0 2024-09-20T19:44:45.056 INFO:tasks.workunit.client.1.smithi160.stdout:5/71: symlink d1/d7/d13/l19 0 2024-09-20T19:44:45.059 INFO:tasks.workunit.client.1.smithi160.stdout:3/74: unlink d0/d6/c8 0 2024-09-20T19:44:45.059 INFO:tasks.workunit.client.1.smithi160.stdout:4/110: dwrite d0/d10/f18 [0,4194304] 0 2024-09-20T19:44:45.059 INFO:tasks.workunit.client.1.smithi160.stdout:4/111: chown d0/d2/d9/f1c 83416 1 2024-09-20T19:44:45.059 INFO:tasks.workunit.client.1.smithi160.stdout:2/121: dread d7/d10/f15 [0,4194304] 0 2024-09-20T19:44:45.063 INFO:tasks.workunit.client.1.smithi160.stdout:9/82: mkdir d0/d8/d16 0 2024-09-20T19:44:45.064 INFO:tasks.workunit.client.1.smithi160.stdout:5/72: unlink d1/d7/f9 0 2024-09-20T19:44:45.065 INFO:tasks.workunit.client.1.smithi160.stdout:3/75: dread d0/d7/fa [0,4194304] 0 2024-09-20T19:44:45.066 INFO:tasks.workunit.client.1.smithi160.stdout:4/112: write d0/d4/dd/f15 [3432673,125486] 0 2024-09-20T19:44:45.066 INFO:tasks.workunit.client.1.smithi160.stdout:4/113: truncate d0/d4/dd/f15 4772903 0 2024-09-20T19:44:45.067 INFO:tasks.workunit.client.1.smithi160.stdout:5/73: dread d1/d7/ff [0,4194304] 0 2024-09-20T19:44:45.074 INFO:tasks.workunit.client.1.smithi160.stdout:2/122: unlink d7/fa 0 2024-09-20T19:44:45.074 INFO:tasks.workunit.client.1.smithi160.stdout:2/123: dread - d7/d10/f20 zero size 2024-09-20T19:44:45.074 INFO:tasks.workunit.client.1.smithi160.stdout:2/124: write d7/fe [1148448,27980] 0 2024-09-20T19:44:45.074 INFO:tasks.workunit.client.1.smithi160.stdout:3/76: dread d0/ff [0,4194304] 0 2024-09-20T19:44:45.078 INFO:tasks.workunit.client.1.smithi160.stdout:9/83: rmdir d0/de 0 2024-09-20T19:44:45.079 INFO:tasks.workunit.client.1.smithi160.stdout:3/77: read d0/ff [466623,74006] 0 2024-09-20T19:44:45.080 INFO:tasks.workunit.client.1.smithi160.stdout:4/114: mknod d0/d2/c23 0 2024-09-20T19:44:45.080 INFO:tasks.workunit.client.1.smithi160.stdout:4/115: stat d0/d11/d22 0 2024-09-20T19:44:45.081 INFO:tasks.workunit.client.1.smithi160.stdout:5/74: mknod d1/d7/d8/c1a 0 2024-09-20T19:44:45.088 INFO:tasks.workunit.client.1.smithi160.stdout:2/125: dwrite d7/f1d [0,4194304] 0 2024-09-20T19:44:45.088 INFO:tasks.workunit.client.1.smithi160.stdout:2/126: dread - d7/d10/f20 zero size 2024-09-20T19:44:45.089 INFO:tasks.workunit.client.1.smithi160.stdout:9/84: rename d0/d4/d13 to d0/d8/d16/d17 0 2024-09-20T19:44:45.097 INFO:tasks.workunit.client.1.smithi160.stdout:2/127: creat d7/d10/d22/d24/f26 x:0 0 0 2024-09-20T19:44:45.097 INFO:tasks.workunit.client.1.smithi160.stdout:2/128: truncate d7/f17 1362583 0 2024-09-20T19:44:45.100 INFO:tasks.workunit.client.1.smithi160.stdout:9/85: creat d0/d4/d5/f18 x:0 0 0 2024-09-20T19:44:45.106 INFO:tasks.workunit.client.1.smithi160.stdout:3/78: dwrite d0/d7/f11 [4194304,4194304] 0 2024-09-20T19:44:45.107 INFO:tasks.workunit.client.1.smithi160.stdout:3/79: chown d0/d7/f11 64204 1 2024-09-20T19:44:45.110 INFO:tasks.workunit.client.1.smithi160.stdout:3/80: mknod d0/d6/c19 0 2024-09-20T19:44:45.111 INFO:tasks.workunit.client.1.smithi160.stdout:3/81: truncate d0/d7/fa 2229965 0 2024-09-20T19:44:45.112 INFO:tasks.workunit.client.1.smithi160.stdout:9/86: dread d0/fa [0,4194304] 0 2024-09-20T19:44:45.112 INFO:tasks.workunit.client.1.smithi160.stdout:9/87: write d0/fc [192766,42145] 0 2024-09-20T19:44:45.113 INFO:tasks.workunit.client.1.smithi160.stdout:3/82: symlink d0/l1a 0 2024-09-20T19:44:45.116 INFO:tasks.workunit.client.1.smithi160.stdout:9/88: unlink d0/d8/d16/d17/l15 0 2024-09-20T19:44:45.117 INFO:tasks.workunit.client.1.smithi160.stdout:3/83: rename d0/c13 to d0/db/c1b 0 2024-09-20T19:44:45.118 INFO:tasks.workunit.client.1.smithi160.stdout:3/84: read d0/db/dc/f15 [251435,44474] 0 2024-09-20T19:44:45.121 INFO:tasks.workunit.client.1.smithi160.stdout:3/85: symlink d0/d6/l1c 0 2024-09-20T19:44:45.121 INFO:tasks.workunit.client.1.smithi160.stdout:3/86: write d0/db/f12 [399686,58585] 0 2024-09-20T19:44:45.121 INFO:tasks.workunit.client.1.smithi160.stdout:3/87: fsync d0/d7/f11 0 2024-09-20T19:44:45.121 INFO:tasks.workunit.client.1.smithi160.stdout:5/75: dwrite f0 [0,4194304] 0 2024-09-20T19:44:45.121 INFO:tasks.workunit.client.1.smithi160.stdout:5/76: write d1/d7/f12 [684525,13776] 0 2024-09-20T19:44:45.121 INFO:tasks.workunit.client.1.smithi160.stdout:5/77: chown d1/d7/d13 1547522199 1 2024-09-20T19:44:45.122 INFO:tasks.workunit.client.1.smithi160.stdout:3/88: read d0/db/f12 [108109,77049] 0 2024-09-20T19:44:45.122 INFO:tasks.workunit.client.1.smithi160.stdout:3/89: write d0/db/f12 [163787,4296] 0 2024-09-20T19:44:45.122 INFO:tasks.workunit.client.1.smithi160.stdout:2/129: dwrite d7/d10/d22/d24/f26 [0,4194304] 0 2024-09-20T19:44:45.122 INFO:tasks.workunit.client.1.smithi160.stdout:4/116: dwrite d0/d2/d9/f1c [0,4194304] 0 2024-09-20T19:44:45.123 INFO:tasks.workunit.client.1.smithi160.stdout:5/78: mknod d1/d7/c1b 0 2024-09-20T19:44:45.123 INFO:tasks.workunit.client.1.smithi160.stdout:0/48: sync 2024-09-20T19:44:45.125 INFO:tasks.workunit.client.1.smithi160.stdout:5/79: dread d1/d7/ff [0,4194304] 0 2024-09-20T19:44:45.126 INFO:tasks.workunit.client.1.smithi160.stdout:3/90: mkdir d0/db/d1d 0 2024-09-20T19:44:45.131 INFO:tasks.workunit.client.1.smithi160.stdout:9/89: dwrite d0/d4/d5/f18 [0,4194304] 0 2024-09-20T19:44:45.132 INFO:tasks.workunit.client.1.smithi160.stdout:9/90: write d0/fc [2848095,68289] 0 2024-09-20T19:44:45.132 INFO:tasks.workunit.client.1.smithi160.stdout:9/91: dread - d0/f14 zero size 2024-09-20T19:44:45.140 INFO:tasks.workunit.client.1.smithi160.stdout:0/49: symlink d4/d6/le 0 2024-09-20T19:44:45.142 INFO:tasks.workunit.client.1.smithi160.stdout:5/80: mknod d1/d7/d13/c1c 0 2024-09-20T19:44:45.142 INFO:tasks.workunit.client.1.smithi160.stdout:5/81: chown d1/d7/l11 129220126 1 2024-09-20T19:44:45.142 INFO:tasks.workunit.client.1.smithi160.stdout:5/82: chown d1/d7/da 247578641 1 2024-09-20T19:44:45.143 INFO:tasks.workunit.client.1.smithi160.stdout:3/91: creat d0/db/d1d/f1e x:0 0 0 2024-09-20T19:44:45.143 INFO:tasks.workunit.client.1.smithi160.stdout:3/92: readlink d0/l3 0 2024-09-20T19:44:45.145 INFO:tasks.workunit.client.1.smithi160.stdout:3/93: dread d0/f2 [0,4194304] 0 2024-09-20T19:44:45.145 INFO:tasks.workunit.client.1.smithi160.stdout:3/94: fsync d0/f2 0 2024-09-20T19:44:45.145 INFO:tasks.workunit.client.1.smithi160.stdout:2/130: rename d7/cb to d7/d10/c27 0 2024-09-20T19:44:45.149 INFO:tasks.workunit.client.1.smithi160.stdout:4/117: dwrite d0/d2/d9/f1c [0,4194304] 0 2024-09-20T19:44:45.149 INFO:tasks.workunit.client.1.smithi160.stdout:9/92: creat d0/f19 x:0 0 0 2024-09-20T19:44:45.149 INFO:tasks.workunit.client.1.smithi160.stdout:9/93: write d0/f14 [401761,118776] 0 2024-09-20T19:44:45.149 INFO:tasks.workunit.client.1.smithi160.stdout:9/94: chown d0/l3 1308 1 2024-09-20T19:44:45.151 INFO:tasks.workunit.client.1.smithi160.stdout:0/50: mkdir d4/d6/df 0 2024-09-20T19:44:45.152 INFO:tasks.workunit.client.1.smithi160.stdout:5/83: unlink d1/d7/l14 0 2024-09-20T19:44:45.156 INFO:tasks.workunit.client.1.smithi160.stdout:3/95: link d0/d6/ce d0/d7/d9/c1f 0 2024-09-20T19:44:45.163 INFO:tasks.workunit.client.1.smithi160.stdout:4/118: rename d0/d2 to d0/d10/d14/d24 0 2024-09-20T19:44:45.163 INFO:tasks.workunit.client.1.smithi160.stdout:4/119: truncate d0/d19/f20 389942 0 2024-09-20T19:44:45.165 INFO:tasks.workunit.client.1.smithi160.stdout:9/95: unlink d0/l10 0 2024-09-20T19:44:45.166 INFO:tasks.workunit.client.1.smithi160.stdout:2/131: dwrite d7/fe [0,4194304] 0 2024-09-20T19:44:45.167 INFO:tasks.workunit.client.1.smithi160.stdout:0/51: creat d4/d6/f10 x:0 0 0 2024-09-20T19:44:45.169 INFO:tasks.workunit.client.1.smithi160.stdout:5/84: symlink d1/l1d 0 2024-09-20T19:44:45.174 INFO:tasks.workunit.client.1.smithi160.stdout:4/120: read d0/d10/d14/d24/d9/f17 [427941,78879] 0 2024-09-20T19:44:45.174 INFO:tasks.workunit.client.1.smithi160.stdout:4/121: fsync d0/d19/f20 0 2024-09-20T19:44:45.174 INFO:tasks.workunit.client.1.smithi160.stdout:4/122: fsync d0/d10/d14/d24/d9/f17 0 2024-09-20T19:44:45.179 INFO:tasks.workunit.client.1.smithi160.stdout:7/38: sync 2024-09-20T19:44:45.179 INFO:tasks.workunit.client.1.smithi160.stdout:7/39: stat d3/d4/f8 0 2024-09-20T19:44:45.179 INFO:tasks.workunit.client.1.smithi160.stdout:7/40: fsync d3/d4/f8 0 2024-09-20T19:44:45.179 INFO:tasks.workunit.client.1.smithi160.stdout:7/41: truncate f1 1532707 0 2024-09-20T19:44:45.179 INFO:tasks.workunit.client.1.smithi160.stdout:7/42: fdatasync d3/d4/f8 0 2024-09-20T19:44:45.181 INFO:tasks.workunit.client.1.smithi160.stdout:2/132: creat d7/d10/f28 x:0 0 0 2024-09-20T19:44:45.181 INFO:tasks.workunit.client.1.smithi160.stdout:2/133: chown d7/d10/l21 28 1 2024-09-20T19:44:45.182 INFO:tasks.workunit.client.1.smithi160.stdout:5/85: rename d1/d7/d13/l19 to d1/d7/d8/l1e 0 2024-09-20T19:44:45.183 INFO:tasks.workunit.client.1.smithi160.stdout:5/86: read d1/f4 [497098,73188] 0 2024-09-20T19:44:45.185 INFO:tasks.workunit.client.1.smithi160.stdout:4/123: mknod d0/d10/d14/d24/d9/c25 0 2024-09-20T19:44:45.188 INFO:tasks.workunit.client.1.smithi160.stdout:7/43: rmdir d3/d4 39 2024-09-20T19:44:45.188 INFO:tasks.workunit.client.1.smithi160.stdout:7/44: write f1 [184078,27038] 0 2024-09-20T19:44:45.189 INFO:tasks.workunit.client.1.smithi160.stdout:5/87: mkdir d1/d7/d8/d1f 0 2024-09-20T19:44:45.189 INFO:tasks.workunit.client.1.smithi160.stdout:9/96: truncate d0/d4/d5/f18 1638645 0 2024-09-20T19:44:45.192 INFO:tasks.workunit.client.1.smithi160.stdout:4/124: symlink d0/d10/d14/d24/d9/l26 0 2024-09-20T19:44:45.193 INFO:tasks.workunit.client.1.smithi160.stdout:7/45: creat d3/d4/fe x:0 0 0 2024-09-20T19:44:45.198 INFO:tasks.workunit.client.1.smithi160.stdout:5/88: truncate d1/f5 2822307 0 2024-09-20T19:44:45.198 INFO:tasks.workunit.client.1.smithi160.stdout:9/97: creat d0/d4/f1a x:0 0 0 2024-09-20T19:44:45.198 INFO:tasks.workunit.client.1.smithi160.stdout:9/98: fsync d0/f14 0 2024-09-20T19:44:45.198 INFO:tasks.workunit.client.1.smithi160.stdout:2/134: dread d7/d10/d22/d24/f26 [0,4194304] 0 2024-09-20T19:44:45.198 INFO:tasks.workunit.client.1.smithi160.stdout:0/52: dwrite d4/d6/fa [0,4194304] 0 2024-09-20T19:44:45.198 INFO:tasks.workunit.client.1.smithi160.stdout:6/78: sync 2024-09-20T19:44:45.198 INFO:tasks.workunit.client.1.smithi160.stdout:1/56: sync 2024-09-20T19:44:45.198 INFO:tasks.workunit.client.1.smithi160.stdout:8/76: sync 2024-09-20T19:44:45.199 INFO:tasks.workunit.client.1.smithi160.stdout:6/79: chown d8/f11 883084 1 2024-09-20T19:44:45.199 INFO:tasks.workunit.client.1.smithi160.stdout:8/77: dread - d1/fb zero size 2024-09-20T19:44:45.199 INFO:tasks.workunit.client.1.smithi160.stdout:4/125: creat d0/d10/d14/d24/f27 x:0 0 0 2024-09-20T19:44:45.199 INFO:tasks.workunit.client.1.smithi160.stdout:6/80: read d8/d9/f15 [713569,27885] 0 2024-09-20T19:44:45.199 INFO:tasks.workunit.client.1.smithi160.stdout:7/46: creat d3/d9/ff x:0 0 0 2024-09-20T19:44:45.202 INFO:tasks.workunit.client.1.smithi160.stdout:7/47: dread d3/d4/f8 [0,4194304] 0 2024-09-20T19:44:45.202 INFO:tasks.workunit.client.1.smithi160.stdout:5/89: mkdir d1/d7/da/d20 0 2024-09-20T19:44:45.205 INFO:tasks.workunit.client.1.smithi160.stdout:2/135: write d7/f1d [1134856,48689] 0 2024-09-20T19:44:45.210 INFO:tasks.workunit.client.1.smithi160.stdout:0/53: symlink d4/d6/l11 0 2024-09-20T19:44:45.212 INFO:tasks.workunit.client.1.smithi160.stdout:8/78: creat d1/f18 x:0 0 0 2024-09-20T19:44:45.212 INFO:tasks.workunit.client.1.smithi160.stdout:8/79: fsync d1/df/d13/f16 0 2024-09-20T19:44:45.212 INFO:tasks.workunit.client.1.smithi160.stdout:8/80: stat d1/cd 0 2024-09-20T19:44:45.215 INFO:tasks.workunit.client.1.smithi160.stdout:4/126: unlink d0/d19/c1f 0 2024-09-20T19:44:45.215 INFO:tasks.workunit.client.1.smithi160.stdout:4/127: chown d0 0 1 2024-09-20T19:44:45.215 INFO:tasks.workunit.client.1.smithi160.stdout:4/128: chown d0/d10/d14/d24/f27 96314726 1 2024-09-20T19:44:45.216 INFO:tasks.workunit.client.1.smithi160.stdout:6/81: creat d8/d9/f19 x:0 0 0 2024-09-20T19:44:45.218 INFO:tasks.workunit.client.1.smithi160.stdout:2/136: dwrite d7/d10/f1e [0,4194304] 0 2024-09-20T19:44:45.219 INFO:tasks.workunit.client.1.smithi160.stdout:7/48: mknod d3/c10 0 2024-09-20T19:44:45.219 INFO:tasks.workunit.client.1.smithi160.stdout:7/49: dread - d3/d9/ff zero size 2024-09-20T19:44:45.220 INFO:tasks.workunit.client.1.smithi160.stdout:5/90: link d1/f4 d1/d7/da/d20/f21 0 2024-09-20T19:44:45.220 INFO:tasks.workunit.client.1.smithi160.stdout:5/91: chown d1/d7/da/f16 3075900 1 2024-09-20T19:44:45.222 INFO:tasks.workunit.client.1.smithi160.stdout:0/54: truncate f0 690373 0 2024-09-20T19:44:45.223 INFO:tasks.workunit.client.1.smithi160.stdout:4/129: mknod d0/d4/c28 0 2024-09-20T19:44:45.223 INFO:tasks.workunit.client.1.smithi160.stdout:4/130: write d0/d19/f20 [837402,32692] 0 2024-09-20T19:44:45.226 INFO:tasks.workunit.client.1.smithi160.stdout:6/82: unlink d8/l17 0 2024-09-20T19:44:45.226 INFO:tasks.workunit.client.1.smithi160.stdout:6/83: dread - d8/d9/f19 zero size 2024-09-20T19:44:45.229 INFO:tasks.workunit.client.1.smithi160.stdout:8/81: dwrite d1/fe [0,4194304] 0 2024-09-20T19:44:45.230 INFO:tasks.workunit.client.1.smithi160.stdout:6/84: dread d8/d9/f12 [0,4194304] 0 2024-09-20T19:44:45.233 INFO:tasks.workunit.client.1.smithi160.stdout:5/92: unlink d1/d7/d8/f18 0 2024-09-20T19:44:45.233 INFO:tasks.workunit.client.1.smithi160.stdout:6/85: dread d8/d9/fa [0,4194304] 0 2024-09-20T19:44:45.236 INFO:tasks.workunit.client.1.smithi160.stdout:4/131: mknod d0/d4/c29 0 2024-09-20T19:44:45.238 INFO:tasks.workunit.client.1.smithi160.stdout:0/55: dwrite d4/d6/f10 [0,4194304] 0 2024-09-20T19:44:45.241 INFO:tasks.workunit.client.1.smithi160.stdout:9/99: rename d0/d4/d5/f18 to d0/d4/f1b 0 2024-09-20T19:44:45.242 INFO:tasks.workunit.client.1.smithi160.stdout:8/82: rename d1/f18 to d1/f19 0 2024-09-20T19:44:45.245 INFO:tasks.workunit.client.1.smithi160.stdout:5/93: symlink d1/d7/l22 0 2024-09-20T19:44:45.247 INFO:tasks.workunit.client.1.smithi160.stdout:6/86: creat d8/f1a x:0 0 0 2024-09-20T19:44:45.249 INFO:tasks.workunit.client.1.smithi160.stdout:4/132: mkdir d0/d19/d2a 0 2024-09-20T19:44:45.253 INFO:tasks.workunit.client.1.smithi160.stdout:0/56: symlink d4/d6/db/l12 0 2024-09-20T19:44:45.253 INFO:tasks.workunit.client.1.smithi160.stdout:0/57: chown f3 1184776895 1 2024-09-20T19:44:45.254 INFO:tasks.workunit.client.1.smithi160.stdout:0/58: stat f0 0 2024-09-20T19:44:45.254 INFO:tasks.workunit.client.1.smithi160.stdout:0/59: chown d4/d6/f10 524 1 2024-09-20T19:44:45.256 INFO:tasks.workunit.client.1.smithi160.stdout:5/94: mkdir d1/d7/da/d23 0 2024-09-20T19:44:45.258 INFO:tasks.workunit.client.1.smithi160.stdout:4/133: dread d0/d10/f18 [0,4194304] 0 2024-09-20T19:44:45.258 INFO:tasks.workunit.client.1.smithi160.stdout:4/134: write d0/d10/d14/d24/f27 [923846,66059] 0 2024-09-20T19:44:45.259 INFO:tasks.workunit.client.1.smithi160.stdout:1/57: rename f3 to d6/da/fe 0 2024-09-20T19:44:45.261 INFO:tasks.workunit.client.1.smithi160.stdout:4/135: creat d0/d11/f2b x:0 0 0 2024-09-20T19:44:45.263 INFO:tasks.workunit.client.1.smithi160.stdout:9/100: rename d0/cf to d0/d8/d16/d17/c1c 0 2024-09-20T19:44:45.266 INFO:tasks.workunit.client.1.smithi160.stdout:1/58: unlink d6/l7 0 2024-09-20T19:44:45.267 INFO:tasks.workunit.client.1.smithi160.stdout:9/101: symlink d0/l1d 0 2024-09-20T19:44:45.267 INFO:tasks.workunit.client.1.smithi160.stdout:9/102: write d0/f14 [1293923,1460] 0 2024-09-20T19:44:45.271 INFO:tasks.workunit.client.1.smithi160.stdout:8/83: rename f0 to d1/df/d13/f1a 0 2024-09-20T19:44:45.271 INFO:tasks.workunit.client.1.smithi160.stdout:8/84: chown d1/f17 115107 1 2024-09-20T19:44:45.273 INFO:tasks.workunit.client.1.smithi160.stdout:1/59: mknod d6/cf 0 2024-09-20T19:44:45.273 INFO:tasks.workunit.client.1.smithi160.stdout:4/136: dread d0/d10/d14/d24/d9/fb [0,4194304] 0 2024-09-20T19:44:45.275 INFO:tasks.workunit.client.1.smithi160.stdout:4/137: write d0/d19/f20 [670668,103573] 0 2024-09-20T19:44:45.275 INFO:tasks.workunit.client.1.smithi160.stdout:2/137: dwrite d7/d10/f1e [4194304,4194304] 0 2024-09-20T19:44:45.275 INFO:tasks.workunit.client.1.smithi160.stdout:4/138: write d0/d19/f20 [983882,45186] 0 2024-09-20T19:44:45.276 INFO:tasks.workunit.client.1.smithi160.stdout:6/87: rename f5 to d8/d9/f1b 0 2024-09-20T19:44:45.276 INFO:tasks.workunit.client.1.smithi160.stdout:6/88: fsync d8/d9/fe 0 2024-09-20T19:44:45.276 INFO:tasks.workunit.client.1.smithi160.stdout:6/89: write d8/d9/f19 [834555,3923] 0 2024-09-20T19:44:45.277 INFO:tasks.workunit.client.1.smithi160.stdout:8/85: mkdir d1/df/d13/d1b 0 2024-09-20T19:44:45.280 INFO:tasks.workunit.client.1.smithi160.stdout:6/90: dread d8/d9/fd [0,4194304] 0 2024-09-20T19:44:45.284 INFO:tasks.workunit.client.1.smithi160.stdout:6/91: dread f1 [0,4194304] 0 2024-09-20T19:44:45.285 INFO:tasks.workunit.client.1.smithi160.stdout:9/103: dwrite d0/d4/f1b [0,4194304] 0 2024-09-20T19:44:45.285 INFO:tasks.workunit.client.1.smithi160.stdout:5/95: rename d1/d7/d13 to d1/d7/d8/d1f/d24 0 2024-09-20T19:44:45.286 INFO:tasks.workunit.client.1.smithi160.stdout:5/96: chown d1/d7/da 1741 1 2024-09-20T19:44:45.287 INFO:tasks.workunit.client.1.smithi160.stdout:8/86: unlink d1/c6 0 2024-09-20T19:44:45.287 INFO:tasks.workunit.client.1.smithi160.stdout:1/60: dread f5 [0,4194304] 0 2024-09-20T19:44:45.289 INFO:tasks.workunit.client.1.smithi160.stdout:2/138: rename d7/d10/d22 to d7/d10/d29 0 2024-09-20T19:44:45.289 INFO:tasks.workunit.client.1.smithi160.stdout:2/139: write d7/d10/f1c [15547,48593] 0 2024-09-20T19:44:45.289 INFO:tasks.workunit.client.1.smithi160.stdout:1/61: write d6/f8 [1108662,55794] 0 2024-09-20T19:44:45.292 INFO:tasks.workunit.client.1.smithi160.stdout:9/104: rename d0/f19 to d0/f1e 0 2024-09-20T19:44:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.291+0000 7fc9e7e2a640 1 -- 172.21.15.114:0/2073383886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e0100d60 msgr2=0x7fc9e0101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.291+0000 7fc9e7e2a640 1 --2- 172.21.15.114:0/2073383886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e0100d60 0x7fc9e0101160 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc009880 tx=0x7fc9dc02a4a0 comp rx=0 tx=0).stop 2024-09-20T19:44:45.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.291+0000 7fc9e7e2a640 1 -- 172.21.15.114:0/2073383886 shutdown_connections 2024-09-20T19:44:45.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.291+0000 7fc9e7e2a640 1 --2- 172.21.15.114:0/2073383886 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9e0101a80 0x7fc9e0101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:45.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.291+0000 7fc9e7e2a640 1 --2- 172.21.15.114:0/2073383886 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e0100d60 0x7fc9e0101160 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:45.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.291+0000 7fc9e7e2a640 1 -- 172.21.15.114:0/2073383886 >> 172.21.15.114:0/2073383886 conn(0x7fc9e00fb750 msgr2=0x7fc9e00fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:45.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.291+0000 7fc9e7e2a640 1 -- 172.21.15.114:0/2073383886 shutdown_connections 2024-09-20T19:44:45.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.291+0000 7fc9e7e2a640 1 -- 172.21.15.114:0/2073383886 wait complete. 2024-09-20T19:44:45.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.292+0000 7fc9e7e2a640 1 Processor -- start 2024-09-20T19:44:45.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.293+0000 7fc9e7e2a640 1 -- start start 2024-09-20T19:44:45.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.293+0000 7fc9e7e2a640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9e0100d60 0x7fc9e0196190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:45.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.293+0000 7fc9e7e2a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e0101a80 0x7fc9e01966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:45.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.293+0000 7fc9e7e2a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e0109670 con 0x7fc9e0101a80 2024-09-20T19:44:45.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.293+0000 7fc9e7e2a640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e01094f0 con 0x7fc9e0100d60 2024-09-20T19:44:45.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.293+0000 7fc9e53c5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e0101a80 0x7fc9e01966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:45.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.293+0000 7fc9e53c5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e0101a80 0x7fc9e01966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49296/0 (socket says 172.21.15.114:49296) 2024-09-20T19:44:45.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.293+0000 7fc9e53c5640 1 -- 172.21.15.114:0/4135587050 learned_addr learned my addr 172.21.15.114:0/4135587050 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:44:45.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.294+0000 7fc9e5bc6640 1 --2- 172.21.15.114:0/4135587050 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9e0100d60 0x7fc9e0196190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:45.297 INFO:tasks.workunit.client.1.smithi160.stdout:5/97: creat d1/d7/d8/f25 x:0 0 0 2024-09-20T19:44:45.297 INFO:tasks.workunit.client.1.smithi160.stdout:5/98: fdatasync d1/d7/d8/fc 0 2024-09-20T19:44:45.297 INFO:tasks.workunit.client.1.smithi160.stdout:5/99: write d1/f5 [3471281,122463] 0 2024-09-20T19:44:45.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.294+0000 7fc9e53c5640 1 -- 172.21.15.114:0/4135587050 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9e0100d60 msgr2=0x7fc9e0196190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:45.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.294+0000 7fc9e53c5640 1 --2- 172.21.15.114:0/4135587050 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9e0100d60 0x7fc9e0196190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:45.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.294+0000 7fc9e53c5640 1 -- 172.21.15.114:0/4135587050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e0196da0 con 0x7fc9e0101a80 2024-09-20T19:44:45.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.294+0000 7fc9e53c5640 1 --2- 172.21.15.114:0/4135587050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e0101a80 0x7fc9e01966d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc9d000e970 tx=0x7fc9d000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:45.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.294+0000 7fc9d6ffd640 1 -- 172.21.15.114:0/4135587050 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc9d000cac0 con 0x7fc9e0101a80 2024-09-20T19:44:45.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.294+0000 7fc9e7e2a640 1 -- 172.21.15.114:0/4135587050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e00ff980 con 0x7fc9e0101a80 2024-09-20T19:44:45.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.295+0000 7fc9e7e2a640 1 -- 172.21.15.114:0/4135587050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9e00ffee0 con 0x7fc9e0101a80 2024-09-20T19:44:45.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.295+0000 7fc9d6ffd640 1 -- 172.21.15.114:0/4135587050 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc9d000cc60 con 0x7fc9e0101a80 2024-09-20T19:44:45.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.295+0000 7fc9d6ffd640 1 -- 172.21.15.114:0/4135587050 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc9d0018720 con 0x7fc9e0101a80 2024-09-20T19:44:45.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.296+0000 7fc9e7e2a640 1 -- 172.21.15.114:0/4135587050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9c4005290 con 0x7fc9e0101a80 2024-09-20T19:44:45.299 INFO:tasks.workunit.client.1.smithi160.stdout:8/87: rename d1/c2 to d1/d14/c1c 0 2024-09-20T19:44:45.299 INFO:tasks.workunit.client.1.smithi160.stdout:8/88: write d1/f19 [512235,47875] 0 2024-09-20T19:44:45.300 INFO:tasks.workunit.client.1.smithi160.stdout:8/89: write d1/f17 [526823,100114] 0 2024-09-20T19:44:45.300 INFO:tasks.workunit.client.1.smithi160.stdout:4/139: dwrite d0/d10/d14/d24/d9/f1c [0,4194304] 0 2024-09-20T19:44:45.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.297+0000 7fc9d6ffd640 1 -- 172.21.15.114:0/4135587050 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fc9d00188e0 con 0x7fc9e0101a80 2024-09-20T19:44:45.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.297+0000 7fc9d6ffd640 1 --2- 172.21.15.114:0/4135587050 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc9cc06e2b0 0x7fc9cc070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:45.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.297+0000 7fc9d6ffd640 1 -- 172.21.15.114:0/4135587050 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7fc9d008e700 con 0x7fc9e0101a80 2024-09-20T19:44:45.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.297+0000 7fc9e5bc6640 1 --2- 172.21.15.114:0/4135587050 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc9cc06e2b0 0x7fc9cc070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:45.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.298+0000 7fc9e5bc6640 1 --2- 172.21.15.114:0/4135587050 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc9cc06e2b0 0x7fc9cc070770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc031460 tx=0x7fc9dc000bc0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:45.301 INFO:tasks.workunit.client.1.smithi160.stdout:2/140: mknod d7/d10/d29/c2a 0 2024-09-20T19:44:45.303 INFO:tasks.workunit.client.1.smithi160.stdout:1/62: creat d6/da/f10 x:0 0 0 2024-09-20T19:44:45.303 INFO:tasks.workunit.client.1.smithi160.stdout:1/63: truncate d6/da/f10 357864 0 2024-09-20T19:44:45.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.303+0000 7fc9d6ffd640 1 -- 172.21.15.114:0/4135587050 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fc9d005fc70 con 0x7fc9e0101a80 2024-09-20T19:44:45.306 INFO:tasks.workunit.client.1.smithi160.stdout:9/105: creat d0/f1f x:0 0 0 2024-09-20T19:44:45.306 INFO:tasks.workunit.client.1.smithi160.stdout:9/106: stat d0/d4 0 2024-09-20T19:44:45.306 INFO:tasks.workunit.client.1.smithi160.stdout:9/107: chown d0/f1f 116194 1 2024-09-20T19:44:45.311 INFO:tasks.workunit.client.1.smithi160.stdout:5/100: rename f0 to d1/d7/da/f26 0 2024-09-20T19:44:45.311 INFO:tasks.workunit.client.1.smithi160.stdout:9/108: rename d0/d8/d16 to d0/d8/d16/d20 22 2024-09-20T19:44:45.314 INFO:tasks.workunit.client.1.smithi160.stdout:8/90: creat d1/d14/f1d x:0 0 0 2024-09-20T19:44:45.314 INFO:tasks.workunit.client.1.smithi160.stdout:8/91: write d1/d14/f1d [928230,27103] 0 2024-09-20T19:44:45.314 INFO:tasks.workunit.client.1.smithi160.stdout:8/92: chown d1/f19 18152 1 2024-09-20T19:44:45.320 INFO:tasks.workunit.client.1.smithi160.stdout:5/101: read d1/d7/d8/fe [3892637,63787] 0 2024-09-20T19:44:45.321 INFO:tasks.workunit.client.1.smithi160.stdout:5/102: dread d1/d7/da/d20/f21 [0,4194304] 0 2024-09-20T19:44:45.321 INFO:tasks.workunit.client.1.smithi160.stdout:5/103: chown d1/d7/da/f16 113816967 1 2024-09-20T19:44:45.322 INFO:tasks.workunit.client.1.smithi160.stdout:6/92: getdents d8/d9 0 2024-09-20T19:44:45.324 INFO:tasks.workunit.client.1.smithi160.stdout:9/109: dwrite d0/d4/f1b [0,4194304] 0 2024-09-20T19:44:45.334 INFO:tasks.workunit.client.1.smithi160.stdout:4/140: rmdir d0/d10/d14 39 2024-09-20T19:44:45.338 INFO:tasks.workunit.client.1.smithi160.stdout:2/141: symlink d7/l2b 0 2024-09-20T19:44:45.339 INFO:tasks.workunit.client.1.smithi160.stdout:1/64: chown d6/da/fb 9 1 2024-09-20T19:44:45.339 INFO:tasks.workunit.client.1.smithi160.stdout:1/65: chown f5 509231991 1 2024-09-20T19:44:45.342 INFO:tasks.workunit.client.1.smithi160.stdout:8/93: symlink d1/l1e 0 2024-09-20T19:44:45.342 INFO:tasks.workunit.client.1.smithi160.stdout:1/66: write d6/da/fb [926021,56512] 0 2024-09-20T19:44:45.343 INFO:tasks.workunit.client.1.smithi160.stdout:5/104: creat d1/d7/d8/f27 x:0 0 0 2024-09-20T19:44:45.345 INFO:tasks.workunit.client.1.smithi160.stdout:6/93: creat d8/f1c x:0 0 0 2024-09-20T19:44:45.345 INFO:tasks.workunit.client.1.smithi160.stdout:6/94: dread - d8/f1c zero size 2024-09-20T19:44:45.346 INFO:tasks.workunit.client.1.smithi160.stdout:9/110: creat d0/d4/f21 x:0 0 0 2024-09-20T19:44:45.348 INFO:tasks.workunit.client.1.smithi160.stdout:6/95: read d8/f11 [496857,13001] 0 2024-09-20T19:44:45.355 INFO:tasks.workunit.client.1.smithi160.stdout:4/141: link d0/d10/d14/d24/d9/c25 d0/c2c 0 2024-09-20T19:44:45.355 INFO:tasks.workunit.client.1.smithi160.stdout:4/142: write d0/f1 [3909659,37882] 0 2024-09-20T19:44:45.355 INFO:tasks.workunit.client.1.smithi160.stdout:2/142: getdents d7/d10 0 2024-09-20T19:44:45.357 INFO:tasks.workunit.client.1.smithi160.stdout:1/67: symlink d6/l11 0 2024-09-20T19:44:45.358 INFO:tasks.workunit.client.1.smithi160.stdout:5/105: creat d1/d7/da/d23/f28 x:0 0 0 2024-09-20T19:44:45.358 INFO:tasks.workunit.client.1.smithi160.stdout:5/106: chown d1/d7 7713 1 2024-09-20T19:44:45.358 INFO:tasks.workunit.client.1.smithi160.stdout:5/107: readlink d1/l1d 0 2024-09-20T19:44:45.361 INFO:tasks.workunit.client.1.smithi160.stdout:8/94: dwrite d1/f19 [0,4194304] 0 2024-09-20T19:44:45.361 INFO:tasks.workunit.client.1.smithi160.stdout:9/111: mknod d0/d8/c22 0 2024-09-20T19:44:45.364 INFO:tasks.workunit.client.1.smithi160.stdout:4/143: truncate d0/d19/f20 344857 0 2024-09-20T19:44:45.365 INFO:tasks.workunit.client.1.smithi160.stdout:2/143: link d7/d10/c27 d7/c2c 0 2024-09-20T19:44:45.365 INFO:tasks.workunit.client.1.smithi160.stdout:2/144: write d7/f1d [3351644,57523] 0 2024-09-20T19:44:45.365 INFO:tasks.workunit.client.1.smithi160.stdout:3/96: sync 2024-09-20T19:44:45.366 INFO:tasks.workunit.client.1.smithi160.stdout:3/97: fdatasync d0/db/d1d/f1e 0 2024-09-20T19:44:45.375 INFO:tasks.workunit.client.1.smithi160.stdout:6/96: getdents d8 0 2024-09-20T19:44:45.376 INFO:tasks.workunit.client.1.smithi160.stdout:8/95: creat d1/df/f1f x:0 0 0 2024-09-20T19:44:45.376 INFO:tasks.workunit.client.1.smithi160.stdout:8/96: read d1/f3 [3278,88858] 0 2024-09-20T19:44:45.377 INFO:tasks.workunit.client.1.smithi160.stdout:9/112: mknod d0/c23 0 2024-09-20T19:44:45.377 INFO:tasks.workunit.client.1.smithi160.stdout:9/113: chown d0/f1f 471329973 1 2024-09-20T19:44:45.377 INFO:tasks.workunit.client.1.smithi160.stdout:9/114: read - d0/d4/f21 zero size 2024-09-20T19:44:45.377 INFO:tasks.workunit.client.1.smithi160.stdout:6/97: dread d8/d9/f12 [0,4194304] 0 2024-09-20T19:44:45.379 INFO:tasks.workunit.client.1.smithi160.stdout:4/144: symlink d0/d10/l2d 0 2024-09-20T19:44:45.379 INFO:tasks.workunit.client.1.smithi160.stdout:2/145: mkdir d7/d10/d29/d24/d2d 0 2024-09-20T19:44:45.380 INFO:tasks.workunit.client.1.smithi160.stdout:7/50: sync 2024-09-20T19:44:45.381 INFO:tasks.workunit.client.1.smithi160.stdout:3/98: creat d0/f20 x:0 0 0 2024-09-20T19:44:45.382 INFO:tasks.workunit.client.1.smithi160.stdout:6/98: mknod d8/d9/c1d 0 2024-09-20T19:44:45.382 INFO:tasks.workunit.client.1.smithi160.stdout:8/97: write d1/df/d13/f1a [415255,124983] 0 2024-09-20T19:44:45.384 INFO:tasks.workunit.client.1.smithi160.stdout:1/68: dwrite d6/f8 [0,4194304] 0 2024-09-20T19:44:45.384 INFO:tasks.workunit.client.1.smithi160.stdout:5/108: dwrite d1/d7/da/d20/f21 [0,4194304] 0 2024-09-20T19:44:45.385 INFO:tasks.workunit.client.1.smithi160.stdout:4/145: symlink d0/d19/d2a/l2e 0 2024-09-20T19:44:45.388 INFO:tasks.workunit.client.1.smithi160.stdout:2/146: link d7/d10/f20 d7/d10/d29/d24/d2d/f2e 0 2024-09-20T19:44:45.391 INFO:tasks.workunit.client.1.smithi160.stdout:6/99: truncate f1 185611 0 2024-09-20T19:44:45.396 INFO:tasks.workunit.client.1.smithi160.stdout:8/98: rename d1/df/d13/f16 to d1/df/f20 0 2024-09-20T19:44:45.396 INFO:tasks.workunit.client.1.smithi160.stdout:8/99: readlink d1/la 0 2024-09-20T19:44:45.402 INFO:tasks.workunit.client.1.smithi160.stdout:1/69: mknod d6/c12 0 2024-09-20T19:44:45.402 INFO:tasks.workunit.client.1.smithi160.stdout:1/70: write d6/da/f10 [790323,98662] 0 2024-09-20T19:44:45.402 INFO:tasks.workunit.client.1.smithi160.stdout:0/60: sync 2024-09-20T19:44:45.402 INFO:tasks.workunit.client.1.smithi160.stdout:0/61: truncate d4/f8 937697 0 2024-09-20T19:44:45.404 INFO:tasks.workunit.client.1.smithi160.stdout:2/147: link d7/d10/d29/d24/d2d/f2e d7/d10/d29/d24/f2f 0 2024-09-20T19:44:45.404 INFO:tasks.workunit.client.1.smithi160.stdout:2/148: truncate d7/fe 4259820 0 2024-09-20T19:44:45.404 INFO:tasks.workunit.client.1.smithi160.stdout:2/149: chown d7/d10/d29/d24/d2d 81063 1 2024-09-20T19:44:45.405 INFO:tasks.workunit.client.1.smithi160.stdout:0/62: rmdir d4/d6/db 39 2024-09-20T19:44:45.407 INFO:tasks.workunit.client.1.smithi160.stdout:2/150: creat d7/d10/d29/f30 x:0 0 0 2024-09-20T19:44:45.407 INFO:tasks.workunit.client.1.smithi160.stdout:0/63: mknod d4/c13 0 2024-09-20T19:44:45.409 INFO:tasks.workunit.client.1.smithi160.stdout:7/51: dwrite d3/d4/fe [0,4194304] 0 2024-09-20T19:44:45.409 INFO:tasks.workunit.client.1.smithi160.stdout:2/151: mkdir d7/d10/d29/d31 0 2024-09-20T19:44:45.409 INFO:tasks.workunit.client.1.smithi160.stdout:2/152: fdatasync d7/d10/f1c 0 2024-09-20T19:44:45.409 INFO:tasks.workunit.client.1.smithi160.stdout:9/115: dwrite d0/f1e [0,4194304] 0 2024-09-20T19:44:45.409 INFO:tasks.workunit.client.1.smithi160.stdout:8/100: dread d1/df/d13/f1a [0,4194304] 0 2024-09-20T19:44:45.410 INFO:tasks.workunit.client.1.smithi160.stdout:3/99: dwrite d0/ff [4194304,4194304] 0 2024-09-20T19:44:45.410 INFO:tasks.workunit.client.1.smithi160.stdout:0/64: mkdir d4/d6/d14 0 2024-09-20T19:44:45.415 INFO:tasks.workunit.client.1.smithi160.stdout:7/52: creat d3/f11 x:0 0 0 2024-09-20T19:44:45.416 INFO:tasks.workunit.client.1.smithi160.stdout:2/153: symlink d7/l32 0 2024-09-20T19:44:45.422 INFO:tasks.workunit.client.1.smithi160.stdout:1/71: dwrite d6/da/f10 [0,4194304] 0 2024-09-20T19:44:45.426 INFO:tasks.workunit.client.1.smithi160.stdout:4/146: rename d0/d19/d2a to d0/d11/d2f 0 2024-09-20T19:44:45.427 INFO:tasks.workunit.client.1.smithi160.stdout:9/116: creat d0/d8/d16/d17/f24 x:0 0 0 2024-09-20T19:44:45.428 INFO:tasks.workunit.client.1.smithi160.stdout:9/117: chown d0/f1f 898624 1 2024-09-20T19:44:45.430 INFO:tasks.workunit.client.1.smithi160.stdout:3/100: getdents d0/d6 0 2024-09-20T19:44:45.432 INFO:tasks.workunit.client.1.smithi160.stdout:0/65: creat d4/f15 x:0 0 0 2024-09-20T19:44:45.433 INFO:tasks.workunit.client.1.smithi160.stdout:7/53: mknod d3/d9/c12 0 2024-09-20T19:44:45.433 INFO:tasks.workunit.client.1.smithi160.stdout:7/54: write d3/f11 [705525,125590] 0 2024-09-20T19:44:45.434 INFO:tasks.workunit.client.1.smithi160.stdout:2/154: dwrite d7/d10/d29/d24/d2d/f2e [0,4194304] 0 2024-09-20T19:44:45.438 INFO:tasks.workunit.client.1.smithi160.stdout:4/147: creat d0/d19/f30 x:0 0 0 2024-09-20T19:44:45.438 INFO:tasks.workunit.client.1.smithi160.stdout:4/148: dread - d0/d19/f30 zero size 2024-09-20T19:44:45.441 INFO:tasks.workunit.client.1.smithi160.stdout:9/118: creat d0/d4/d5/f25 x:0 0 0 2024-09-20T19:44:45.443 INFO:tasks.workunit.client.1.smithi160.stdout:3/101: creat d0/f21 x:0 0 0 2024-09-20T19:44:45.444 INFO:tasks.workunit.client.1.smithi160.stdout:0/66: creat d4/d6/db/f16 x:0 0 0 2024-09-20T19:44:45.448 INFO:tasks.workunit.client.1.smithi160.stdout:2/155: creat d7/d10/f33 x:0 0 0 2024-09-20T19:44:45.449 INFO:tasks.workunit.client.1.smithi160.stdout:1/72: mkdir d6/da/d13 0 2024-09-20T19:44:45.451 INFO:tasks.workunit.client.1.smithi160.stdout:7/55: dwrite d3/d4/f8 [0,4194304] 0 2024-09-20T19:44:45.451 INFO:tasks.workunit.client.1.smithi160.stdout:6/100: rename d8/d9/fa to d8/f1e 0 2024-09-20T19:44:45.453 INFO:tasks.workunit.client.1.smithi160.stdout:9/119: creat d0/d4/d5/f26 x:0 0 0 2024-09-20T19:44:45.453 INFO:tasks.workunit.client.1.smithi160.stdout:9/120: truncate d0/d4/f1b 4686803 0 2024-09-20T19:44:45.454 INFO:tasks.workunit.client.1.smithi160.stdout:3/102: creat d0/db/dc/f22 x:0 0 0 2024-09-20T19:44:45.454 INFO:tasks.workunit.client.1.smithi160.stdout:0/67: dread d4/d6/f10 [0,4194304] 0 2024-09-20T19:44:45.455 INFO:tasks.workunit.client.1.smithi160.stdout:2/156: mkdir d7/d10/d34 0 2024-09-20T19:44:45.456 INFO:tasks.workunit.client.1.smithi160.stdout:3/103: dread d0/f2 [0,4194304] 0 2024-09-20T19:44:45.459 INFO:tasks.workunit.client.1.smithi160.stdout:7/56: link d3/d9/ca d3/d4/c13 0 2024-09-20T19:44:45.460 INFO:tasks.workunit.client.1.smithi160.stdout:4/149: rename d0/d4/dd to d0/d31 0 2024-09-20T19:44:45.460 INFO:tasks.workunit.client.1.smithi160.stdout:4/150: readlink d0/d10/d14/d24/d9/l26 0 2024-09-20T19:44:45.462 INFO:tasks.workunit.client.1.smithi160.stdout:6/101: creat d8/f1f x:0 0 0 2024-09-20T19:44:45.465 INFO:tasks.workunit.client.1.smithi160.stdout:9/121: creat d0/d8/d16/f27 x:0 0 0 2024-09-20T19:44:45.465 INFO:tasks.workunit.client.1.smithi160.stdout:9/122: write d0/d4/d5/f25 [289970,53908] 0 2024-09-20T19:44:45.465 INFO:tasks.workunit.client.1.smithi160.stdout:9/123: chown d0/l6 337200 1 2024-09-20T19:44:45.465 INFO:tasks.workunit.client.1.smithi160.stdout:9/124: write d0/d8/d16/d17/f24 [275,101466] 0 2024-09-20T19:44:45.467 INFO:tasks.workunit.client.1.smithi160.stdout:3/104: dread d0/ff [4194304,4194304] 0 2024-09-20T19:44:45.467 INFO:tasks.workunit.client.1.smithi160.stdout:3/105: dread - d0/f21 zero size 2024-09-20T19:44:45.467 INFO:tasks.workunit.client.1.smithi160.stdout:3/106: read - d0/db/d1d/f1e zero size 2024-09-20T19:44:45.467 INFO:tasks.workunit.client.1.smithi160.stdout:2/157: mknod d7/d10/c35 0 2024-09-20T19:44:45.467 INFO:tasks.workunit.client.1.smithi160.stdout:2/158: truncate d7/f25 644514 0 2024-09-20T19:44:45.468 INFO:tasks.workunit.client.1.smithi160.stdout:7/57: dread d3/d4/f8 [0,4194304] 0 2024-09-20T19:44:45.468 INFO:tasks.workunit.client.1.smithi160.stdout:7/58: truncate d3/d4/fe 5118363 0 2024-09-20T19:44:45.469 INFO:tasks.workunit.client.1.smithi160.stdout:2/159: dread d7/f17 [0,4194304] 0 2024-09-20T19:44:45.469 INFO:tasks.workunit.client.1.smithi160.stdout:0/68: dwrite d4/d6/db/f16 [0,4194304] 0 2024-09-20T19:44:45.470 INFO:tasks.workunit.client.1.smithi160.stdout:4/151: mkdir d0/d19/d32 0 2024-09-20T19:44:45.472 INFO:tasks.workunit.client.1.smithi160.stdout:9/125: creat d0/d8/f28 x:0 0 0 2024-09-20T19:44:45.475 INFO:tasks.workunit.client.1.smithi160.stdout:1/73: rename d6/cf to d6/da/c14 0 2024-09-20T19:44:45.475 INFO:tasks.workunit.client.1.smithi160.stdout:1/74: stat d6/da 0 2024-09-20T19:44:45.476 INFO:tasks.workunit.client.1.smithi160.stdout:7/59: creat d3/f14 x:0 0 0 2024-09-20T19:44:45.478 INFO:tasks.workunit.client.1.smithi160.stdout:2/160: mkdir d7/d10/d34/d36 0 2024-09-20T19:44:45.484 INFO:tasks.workunit.client.1.smithi160.stdout:1/75: dread d6/da/fe [0,4194304] 0 2024-09-20T19:44:45.488 INFO:tasks.workunit.client.1.smithi160.stdout:1/76: fdatasync d6/da/fb 0 2024-09-20T19:44:45.488 INFO:tasks.workunit.client.1.smithi160.stdout:1/77: fsync d6/da/f10 0 2024-09-20T19:44:45.488 INFO:tasks.workunit.client.1.smithi160.stdout:1/78: getdents d6/da/d13 0 2024-09-20T19:44:45.489 INFO:tasks.workunit.client.1.smithi160.stdout:4/152: symlink d0/d4/l33 0 2024-09-20T19:44:45.495 INFO:tasks.workunit.client.1.smithi160.stdout:9/126: mknod d0/d4/c29 0 2024-09-20T19:44:45.495 INFO:tasks.workunit.client.1.smithi160.stdout:9/127: chown d0/f1f 339353 1 2024-09-20T19:44:45.495 INFO:tasks.workunit.client.1.smithi160.stdout:9/128: chown d0/d8 404426475 1 2024-09-20T19:44:45.497 INFO:tasks.workunit.client.1.smithi160.stdout:2/161: dwrite d7/d10/f16 [0,4194304] 0 2024-09-20T19:44:45.497 INFO:tasks.workunit.client.1.smithi160.stdout:3/107: rename d0/db/c16 to d0/d7/d9/c23 0 2024-09-20T19:44:45.497 INFO:tasks.workunit.client.1.smithi160.stdout:3/108: read d0/d7/fa [834185,92175] 0 2024-09-20T19:44:45.499 INFO:tasks.workunit.client.1.smithi160.stdout:7/60: truncate f0 126544 0 2024-09-20T19:44:45.499 INFO:tasks.workunit.client.1.smithi160.stdout:1/79: mknod d6/da/d13/c15 0 2024-09-20T19:44:45.503 INFO:tasks.workunit.client.1.smithi160.stdout:6/102: fsync d8/f1f 0 2024-09-20T19:44:45.504 INFO:tasks.workunit.client.1.smithi160.stdout:6/103: write d8/f1f [660166,30337] 0 2024-09-20T19:44:45.504 INFO:tasks.workunit.client.1.smithi160.stdout:1/80: read f5 [233096,73389] 0 2024-09-20T19:44:45.505 INFO:tasks.workunit.client.1.smithi160.stdout:9/129: dread d0/fc [0,4194304] 0 2024-09-20T19:44:45.505 INFO:tasks.workunit.client.1.smithi160.stdout:9/130: dread - d0/d8/d16/f27 zero size 2024-09-20T19:44:45.506 INFO:tasks.workunit.client.1.smithi160.stdout:2/162: mkdir d7/d10/d34/d37 0 2024-09-20T19:44:45.506 INFO:tasks.workunit.client.1.smithi160.stdout:7/61: dread d3/d4/f8 [0,4194304] 0 2024-09-20T19:44:45.507 INFO:tasks.workunit.client.1.smithi160.stdout:3/109: mkdir d0/d6/d24 0 2024-09-20T19:44:45.508 INFO:tasks.workunit.client.1.smithi160.stdout:4/153: dwrite d0/f1 [4194304,4194304] 0 2024-09-20T19:44:45.509 INFO:tasks.workunit.client.1.smithi160.stdout:6/104: unlink d8/c13 0 2024-09-20T19:44:45.513 INFO:tasks.workunit.client.1.smithi160.stdout:9/131: creat d0/d4/d5/f2a x:0 0 0 2024-09-20T19:44:45.513 INFO:tasks.workunit.client.1.smithi160.stdout:2/163: dread d7/d10/f15 [0,4194304] 0 2024-09-20T19:44:45.513 INFO:tasks.workunit.client.1.smithi160.stdout:7/62: symlink d3/d9/l15 0 2024-09-20T19:44:45.516 INFO:tasks.workunit.client.1.smithi160.stdout:4/154: creat d0/f34 x:0 0 0 2024-09-20T19:44:45.516 INFO:tasks.workunit.client.1.smithi160.stdout:4/155: readlink d0/d10/l2d 0 2024-09-20T19:44:45.517 INFO:tasks.workunit.client.1.smithi160.stdout:1/81: write d6/da/f10 [2226203,58612] 0 2024-09-20T19:44:45.517 INFO:tasks.workunit.client.1.smithi160.stdout:1/82: chown d6/da/d13 112565758 1 2024-09-20T19:44:45.517 INFO:tasks.workunit.client.1.smithi160.stdout:1/83: fdatasync f5 0 2024-09-20T19:44:45.519 INFO:tasks.workunit.client.1.smithi160.stdout:6/105: rmdir d8 39 2024-09-20T19:44:45.519 INFO:tasks.workunit.client.1.smithi160.stdout:6/106: read d8/d9/f1b [100480,3040] 0 2024-09-20T19:44:45.520 INFO:tasks.workunit.client.1.smithi160.stdout:7/63: creat d3/d4/f16 x:0 0 0 2024-09-20T19:44:45.522 INFO:tasks.workunit.client.1.smithi160.stdout:9/132: mknod d0/d8/d16/d17/c2b 0 2024-09-20T19:44:45.522 INFO:tasks.workunit.client.1.smithi160.stdout:9/133: fdatasync d0/f1e 0 2024-09-20T19:44:45.523 INFO:tasks.workunit.client.1.smithi160.stdout:3/110: dwrite d0/d7/f11 [0,4194304] 0 2024-09-20T19:44:45.523 INFO:tasks.workunit.client.1.smithi160.stdout:3/111: dread - d0/db/d1d/f1e zero size 2024-09-20T19:44:45.523 INFO:tasks.workunit.client.1.smithi160.stdout:3/112: chown d0/db 2940552 1 2024-09-20T19:44:45.523 INFO:tasks.workunit.client.1.smithi160.stdout:3/113: read - d0/f21 zero size 2024-09-20T19:44:45.523 INFO:tasks.workunit.client.1.smithi160.stdout:3/114: truncate d0/db/dc/f15 1466746 0 2024-09-20T19:44:45.523 INFO:tasks.workunit.client.1.smithi160.stdout:2/164: dread d7/d10/d29/d24/d2d/f2e [0,4194304] 0 2024-09-20T19:44:45.525 INFO:tasks.workunit.client.1.smithi160.stdout:1/84: creat d6/da/f16 x:0 0 0 2024-09-20T19:44:45.526 INFO:tasks.workunit.client.1.smithi160.stdout:9/134: link d0/c23 d0/d4/c2c 0 2024-09-20T19:44:45.526 INFO:tasks.workunit.client.1.smithi160.stdout:9/135: dread - d0/d8/d16/f27 zero size 2024-09-20T19:44:45.529 INFO:tasks.workunit.client.1.smithi160.stdout:3/115: mkdir d0/db/d25 0 2024-09-20T19:44:45.533 INFO:tasks.workunit.client.1.smithi160.stdout:9/136: creat d0/d8/d16/f2d x:0 0 0 2024-09-20T19:44:45.535 INFO:tasks.workunit.client.1.smithi160.stdout:3/116: rename d0/f21 to d0/db/f26 0 2024-09-20T19:44:45.537 INFO:tasks.workunit.client.1.smithi160.stdout:6/107: dwrite d8/f11 [0,4194304] 0 2024-09-20T19:44:45.537 INFO:tasks.workunit.client.1.smithi160.stdout:7/64: write f0 [1046385,61945] 0 2024-09-20T19:44:45.539 INFO:tasks.workunit.client.1.smithi160.stdout:1/85: symlink d6/da/l17 0 2024-09-20T19:44:45.539 INFO:tasks.workunit.client.1.smithi160.stdout:5/109: sync 2024-09-20T19:44:45.539 INFO:tasks.workunit.client.1.smithi160.stdout:5/110: fsync d1/d7/d8/f25 0 2024-09-20T19:44:45.545 INFO:tasks.workunit.client.1.smithi160.stdout:6/108: mknod d8/c20 0 2024-09-20T19:44:45.546 INFO:tasks.workunit.client.1.smithi160.stdout:6/109: write d8/d9/f15 [1924953,66911] 0 2024-09-20T19:44:45.546 INFO:tasks.workunit.client.1.smithi160.stdout:6/110: write d8/f1a [550178,47943] 0 2024-09-20T19:44:45.548 INFO:tasks.workunit.client.1.smithi160.stdout:2/165: dwrite d7/fe [0,4194304] 0 2024-09-20T19:44:45.549 INFO:tasks.workunit.client.1.smithi160.stdout:1/86: rename d6/da/l17 to d6/da/d13/l18 0 2024-09-20T19:44:45.549 INFO:tasks.workunit.client.1.smithi160.stdout:1/87: chown d6/da/fe 591328 1 2024-09-20T19:44:45.549 INFO:tasks.workunit.client.1.smithi160.stdout:1/88: fdatasync d6/da/f10 0 2024-09-20T19:44:45.551 INFO:tasks.workunit.client.1.smithi160.stdout:6/111: rename d8/f11 to d8/f21 0 2024-09-20T19:44:45.553 INFO:tasks.workunit.client.1.smithi160.stdout:1/89: dread f5 [0,4194304] 0 2024-09-20T19:44:45.553 INFO:tasks.workunit.client.1.smithi160.stdout:6/112: write d8/d9/fe [730040,56140] 0 2024-09-20T19:44:45.555 INFO:tasks.workunit.client.1.smithi160.stdout:2/166: symlink d7/d10/d34/d36/l38 0 2024-09-20T19:44:45.558 INFO:tasks.workunit.client.1.smithi160.stdout:1/90: rmdir d6 39 2024-09-20T19:44:45.563 INFO:tasks.workunit.client.1.smithi160.stdout:6/113: mkdir d8/d9/d22 0 2024-09-20T19:44:45.563 INFO:tasks.workunit.client.1.smithi160.stdout:2/167: write d7/d10/f15 [2339599,82220] 0 2024-09-20T19:44:45.568 INFO:tasks.workunit.client.1.smithi160.stdout:2/168: symlink d7/d10/d34/l39 0 2024-09-20T19:44:45.568 INFO:tasks.workunit.client.1.smithi160.stdout:2/169: readlink d7/l2b 0 2024-09-20T19:44:45.569 INFO:tasks.workunit.client.1.smithi160.stdout:2/170: chown d7/d10/d34 406668 1 2024-09-20T19:44:45.576 INFO:tasks.workunit.client.1.smithi160.stdout:7/65: dwrite d3/d4/f16 [0,4194304] 0 2024-09-20T19:44:45.580 INFO:tasks.workunit.client.1.smithi160.stdout:3/117: rename d0/db to d0/d27 0 2024-09-20T19:44:45.582 INFO:tasks.workunit.client.1.smithi160.stdout:7/66: write d3/d4/f8 [1813624,87830] 0 2024-09-20T19:44:45.582 INFO:tasks.workunit.client.1.smithi160.stdout:7/67: truncate d3/d4/f16 5173074 0 2024-09-20T19:44:45.583 INFO:tasks.workunit.client.1.smithi160.stdout:5/111: dwrite d1/d7/da/f26 [0,4194304] 0 2024-09-20T19:44:45.583 INFO:tasks.workunit.client.1.smithi160.stdout:5/112: write d1/d7/da/d23/f28 [688942,104400] 0 2024-09-20T19:44:45.584 INFO:tasks.workunit.client.1.smithi160.stdout:3/118: mknod d0/d27/c28 0 2024-09-20T19:44:45.587 INFO:tasks.workunit.client.1.smithi160.stdout:7/68: unlink d3/d4/f8 0 2024-09-20T19:44:45.589 INFO:tasks.workunit.client.1.smithi160.stdout:3/119: creat d0/d7/f29 x:0 0 0 2024-09-20T19:44:45.589 INFO:tasks.workunit.client.1.smithi160.stdout:7/69: symlink d3/d9/l17 0 2024-09-20T19:44:45.591 INFO:tasks.workunit.client.1.smithi160.stdout:9/137: dwrite d0/fc [0,4194304] 0 2024-09-20T19:44:45.591 INFO:tasks.workunit.client.1.smithi160.stdout:9/138: stat d0/d8/d16 0 2024-09-20T19:44:45.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.594+0000 7fc9e7e2a640 1 -- 172.21.15.114:0/4135587050 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7fc9c4000bc0 con 0x7fc9cc06e2b0 2024-09-20T19:44:45.596 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:44:45 smithi114 ceph-mon[28260]: pgmap v238: 65 pgs: 65 active+clean; 195 MiB data, 898 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 9.8 MiB/s wr, 892 op/s 2024-09-20T19:44:45.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.596+0000 7fc9d6ffd640 1 -- 172.21.15.114:0/4135587050 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+439 (secure 0 0 0) 0x7fc9c4000bc0 con 0x7fc9cc06e2b0 2024-09-20T19:44:45.600 INFO:tasks.workunit.client.1.smithi160.stdout:3/120: dread d0/d7/f11 [0,4194304] 0 2024-09-20T19:44:45.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.599+0000 7fc9e7e2a640 1 -- 172.21.15.114:0/4135587050 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc9cc06e2b0 msgr2=0x7fc9cc070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:45.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.599+0000 7fc9e7e2a640 1 --2- 172.21.15.114:0/4135587050 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc9cc06e2b0 0x7fc9cc070770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc031460 tx=0x7fc9dc000bc0 comp rx=0 tx=0).stop 2024-09-20T19:44:45.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.600+0000 7fc9e7e2a640 1 -- 172.21.15.114:0/4135587050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e0101a80 msgr2=0x7fc9e01966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:45.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.600+0000 7fc9e7e2a640 1 --2- 172.21.15.114:0/4135587050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e0101a80 0x7fc9e01966d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc9d000e970 tx=0x7fc9d000ee30 comp rx=0 tx=0).stop 2024-09-20T19:44:45.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.600+0000 7fc9e7e2a640 1 -- 172.21.15.114:0/4135587050 shutdown_connections 2024-09-20T19:44:45.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.600+0000 7fc9e7e2a640 1 --2- 172.21.15.114:0/4135587050 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fc9cc06e2b0 0x7fc9cc070770 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:45.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.600+0000 7fc9e7e2a640 1 --2- 172.21.15.114:0/4135587050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e0101a80 0x7fc9e01966d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:45.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.600+0000 7fc9e7e2a640 1 --2- 172.21.15.114:0/4135587050 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc9e0100d60 0x7fc9e0196190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:45.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.600+0000 7fc9e7e2a640 1 -- 172.21.15.114:0/4135587050 >> 172.21.15.114:0/4135587050 conn(0x7fc9e00fb750 msgr2=0x7fc9e0104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:45.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.600+0000 7fc9e7e2a640 1 -- 172.21.15.114:0/4135587050 shutdown_connections 2024-09-20T19:44:45.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.600+0000 7fc9e7e2a640 1 -- 172.21.15.114:0/4135587050 wait complete. 2024-09-20T19:44:45.606 INFO:tasks.workunit.client.1.smithi160.stdout:3/121: creat d0/d6/f2a x:0 0 0 2024-09-20T19:44:45.606 INFO:tasks.workunit.client.1.smithi160.stdout:3/122: write d0/d6/f2a [280048,13348] 0 2024-09-20T19:44:45.606 INFO:tasks.workunit.client.1.smithi160.stdout:3/123: chown d0/ld 2841 1 2024-09-20T19:44:45.606 INFO:tasks.workunit.client.1.smithi160.stdout:3/124: dread - d0/f20 zero size 2024-09-20T19:44:45.607 INFO:tasks.workunit.client.1.smithi160.stdout:3/125: rename d0/d27/d25 to d0/d27/d2b 0 2024-09-20T19:44:45.610 INFO:tasks.workunit.client.1.smithi160.stdout:1/91: dwrite f5 [0,4194304] 0 2024-09-20T19:44:45.610 INFO:tasks.workunit.client.1.smithi160.stdout:6/114: dwrite d8/d9/f19 [0,4194304] 0 2024-09-20T19:44:45.611 INFO:tasks.workunit.client.1.smithi160.stdout:2/171: dwrite d7/d10/f33 [0,4194304] 0 2024-09-20T19:44:45.613 INFO:tasks.workunit.client.1.smithi160.stdout:9/139: dwrite d0/f14 [0,4194304] 0 2024-09-20T19:44:45.613 INFO:tasks.workunit.client.1.smithi160.stdout:9/140: dread - d0/d4/f1a zero size 2024-09-20T19:44:45.613 INFO:tasks.workunit.client.1.smithi160.stdout:9/141: stat d0/d8/d16/d17 0 2024-09-20T19:44:45.613 INFO:tasks.workunit.client.1.smithi160.stdout:9/142: chown d0 9204 1 2024-09-20T19:44:45.613 INFO:tasks.workunit.client.1.smithi160.stdout:7/70: dwrite f1 [0,4194304] 0 2024-09-20T19:44:45.613 INFO:tasks.workunit.client.1.smithi160.stdout:7/71: dread - d3/f14 zero size 2024-09-20T19:44:45.614 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-20T19:44:45.620 INFO:tasks.workunit.client.1.smithi160.stdout:6/115: rmdir d8 39 2024-09-20T19:44:45.627 INFO:tasks.workunit.client.1.smithi160.stdout:3/126: dwrite d0/d27/d1d/f1e [0,4194304] 0 2024-09-20T19:44:45.627 INFO:tasks.workunit.client.1.smithi160.stdout:5/113: dwrite d1/d7/da/d23/f28 [0,4194304] 0 2024-09-20T19:44:45.631 INFO:tasks.workunit.client.1.smithi160.stdout:3/127: write d0/d7/f11 [3299083,60850] 0 2024-09-20T19:44:45.633 INFO:tasks.workunit.client.1.smithi160.stdout:5/114: dread d1/d7/d8/fe [0,4194304] 0 2024-09-20T19:44:45.636 INFO:tasks.workunit.client.1.smithi160.stdout:2/172: symlink d7/d10/d29/l3a 0 2024-09-20T19:44:45.641 INFO:tasks.workunit.client.1.smithi160.stdout:2/173: write d7/d10/f20 [247276,20221] 0 2024-09-20T19:44:45.641 INFO:tasks.workunit.client.1.smithi160.stdout:9/143: rename d0/fa to d0/f2e 0 2024-09-20T19:44:45.647 INFO:tasks.workunit.client.1.smithi160.stdout:7/72: mknod d3/c18 0 2024-09-20T19:44:45.649 INFO:tasks.workunit.client.1.smithi160.stdout:1/92: rename f5 to d6/da/f19 0 2024-09-20T19:44:45.650 INFO:tasks.workunit.client.1.smithi160.stdout:1/93: chown d6/da/d13/c15 46778 1 2024-09-20T19:44:45.650 INFO:tasks.workunit.client.1.smithi160.stdout:6/116: mknod d8/c23 0 2024-09-20T19:44:45.650 INFO:tasks.workunit.client.1.smithi160.stdout:6/117: stat d8/c18 0 2024-09-20T19:44:45.651 INFO:tasks.workunit.client.1.smithi160.stdout:6/118: chown d8/f1a 934060053 1 2024-09-20T19:44:45.656 INFO:tasks.workunit.client.1.smithi160.stdout:7/73: dread f1 [0,4194304] 0 2024-09-20T19:44:45.656 INFO:tasks.workunit.client.1.smithi160.stdout:7/74: chown d3/c18 150 1 2024-09-20T19:44:45.657 INFO:tasks.workunit.client.1.smithi160.stdout:5/115: symlink d1/d7/d8/d1f/l29 0 2024-09-20T19:44:45.658 INFO:tasks.workunit.client.1.smithi160.stdout:6/119: dread d8/d9/f15 [0,4194304] 0 2024-09-20T19:44:45.661 INFO:tasks.workunit.client.1.smithi160.stdout:2/174: rmdir d7/d10/d34 39 2024-09-20T19:44:45.661 INFO:tasks.workunit.client.1.smithi160.stdout:6/120: dread d8/d9/f15 [0,4194304] 0 2024-09-20T19:44:45.662 INFO:tasks.workunit.client.1.smithi160.stdout:9/144: mknod d0/c2f 0 2024-09-20T19:44:45.662 INFO:tasks.workunit.client.1.smithi160.stdout:9/145: write d0/d8/f28 [637861,13740] 0 2024-09-20T19:44:45.662 INFO:tasks.workunit.client.1.smithi160.stdout:9/146: dread - d0/f1f zero size 2024-09-20T19:44:45.662 INFO:tasks.workunit.client.1.smithi160.stdout:3/128: fsync d0/d6/f2a 0 2024-09-20T19:44:45.662 INFO:tasks.workunit.client.1.smithi160.stdout:3/129: read - d0/d7/f29 zero size 2024-09-20T19:44:45.666 INFO:tasks.workunit.client.1.smithi160.stdout:1/94: truncate d6/f8 2051042 0 2024-09-20T19:44:45.668 INFO:tasks.workunit.client.1.smithi160.stdout:7/75: rename d3/f11 to d3/d9/f19 0 2024-09-20T19:44:45.670 INFO:tasks.workunit.client.1.smithi160.stdout:5/116: unlink d1/d7/d8/c1a 0 2024-09-20T19:44:45.671 INFO:tasks.workunit.client.1.smithi160.stdout:2/175: symlink d7/d10/d34/l3b 0 2024-09-20T19:44:45.671 INFO:tasks.workunit.client.1.smithi160.stdout:2/176: stat d7/f1f 0 2024-09-20T19:44:45.671 INFO:tasks.workunit.client.1.smithi160.stdout:2/177: fsync d7/f17 0 2024-09-20T19:44:45.671 INFO:tasks.workunit.client.1.smithi160.stdout:2/178: chown d7/l2b 2 1 2024-09-20T19:44:45.671 INFO:tasks.workunit.client.1.smithi160.stdout:2/179: write d7/d10/f15 [5236479,12456] 0 2024-09-20T19:44:45.674 INFO:tasks.workunit.client.1.smithi160.stdout:6/121: creat d8/d9/d22/f24 x:0 0 0 2024-09-20T19:44:45.674 INFO:tasks.workunit.client.1.smithi160.stdout:6/122: chown d8/d9/fe 4341 1 2024-09-20T19:44:45.674 INFO:tasks.workunit.client.1.smithi160.stdout:6/123: readlink d8/l14 0 2024-09-20T19:44:45.674 INFO:tasks.workunit.client.1.smithi160.stdout:9/147: rmdir d0/d8/d16/d17 39 2024-09-20T19:44:45.679 INFO:tasks.workunit.client.1.smithi160.stdout:7/76: rmdir d3 39 2024-09-20T19:44:45.680 INFO:tasks.workunit.client.1.smithi160.stdout:7/77: fsync f0 0 2024-09-20T19:44:45.680 INFO:tasks.workunit.client.1.smithi160.stdout:7/78: dread - d3/d9/ff zero size 2024-09-20T19:44:45.680 INFO:tasks.workunit.client.1.smithi160.stdout:7/79: readlink d3/l6 0 2024-09-20T19:44:45.683 INFO:tasks.workunit.client.1.smithi160.stdout:6/124: symlink d8/d9/d22/l25 0 2024-09-20T19:44:45.687 INFO:tasks.workunit.client.1.smithi160.stdout:7/80: mknod d3/d4/c1a 0 2024-09-20T19:44:45.687 INFO:tasks.workunit.client.1.smithi160.stdout:7/81: write d3/d9/f19 [1761557,54896] 0 2024-09-20T19:44:45.690 INFO:tasks.workunit.client.1.smithi160.stdout:6/125: link d8/d9/f15 d8/d9/d22/f26 0 2024-09-20T19:44:45.691 INFO:tasks.workunit.client.1.smithi160.stdout:7/82: mkdir d3/d9/d1b 0 2024-09-20T19:44:45.693 INFO:tasks.workunit.client.1.smithi160.stdout:7/83: rename d3/d9/ff to d3/d9/f1c 0 2024-09-20T19:44:45.696 INFO:tasks.workunit.client.1.smithi160.stdout:7/84: creat d3/d4/f1d x:0 0 0 2024-09-20T19:44:45.698 INFO:tasks.workunit.client.1.smithi160.stdout:7/85: rmdir d3/d9/d1b 0 2024-09-20T19:44:45.700 INFO:tasks.workunit.client.1.smithi160.stdout:7/86: creat d3/d4/f1e x:0 0 0 2024-09-20T19:44:45.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.711+0000 7f73fe907640 1 -- 172.21.15.114:0/3711543689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f81023d0 msgr2=0x7f73f81047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:45.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.711+0000 7f73fe907640 1 --2- 172.21.15.114:0/3711543689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f81023d0 0x7f73f81047c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f73e80098b0 tx=0x7f73e802a4d0 comp rx=0 tx=0).stop 2024-09-20T19:44:45.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.711+0000 7f73fe907640 1 -- 172.21.15.114:0/3711543689 shutdown_connections 2024-09-20T19:44:45.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.711+0000 7f73fe907640 1 --2- 172.21.15.114:0/3711543689 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f73f8104d00 0x7f73f81070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:45.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.711+0000 7f73fe907640 1 --2- 172.21.15.114:0/3711543689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f81023d0 0x7f73f81047c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:45.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.711+0000 7f73fe907640 1 -- 172.21.15.114:0/3711543689 >> 172.21.15.114:0/3711543689 conn(0x7f73f80fbff0 msgr2=0x7f73f80fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:45.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.712+0000 7f73fe907640 1 -- 172.21.15.114:0/3711543689 shutdown_connections 2024-09-20T19:44:45.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.712+0000 7f73fe907640 1 -- 172.21.15.114:0/3711543689 wait complete. 2024-09-20T19:44:45.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.712+0000 7f73fe907640 1 Processor -- start 2024-09-20T19:44:45.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.712+0000 7f73fe907640 1 -- start start 2024-09-20T19:44:45.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.713+0000 7f73fe907640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f73f81023d0 0x7f73f8101cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:45.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.713+0000 7f73fe907640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f8104d00 0x7f73f8100380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:45.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.713+0000 7f73fe907640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f73f810a7c0 con 0x7f73f8104d00 2024-09-20T19:44:45.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.713+0000 7f73fe907640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f73f810a640 con 0x7f73f81023d0 2024-09-20T19:44:45.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.713+0000 7f73f77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f8104d00 0x7f73f8100380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:45.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.713+0000 7f73f77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f8104d00 0x7f73f8100380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49310/0 (socket says 172.21.15.114:49310) 2024-09-20T19:44:45.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.713+0000 7f73f77fe640 1 -- 172.21.15.114:0/4281504079 learned_addr learned my addr 172.21.15.114:0/4281504079 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:44:45.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.713+0000 7f73f7fff640 1 --2- 172.21.15.114:0/4281504079 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f73f81023d0 0x7f73f8101cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:45.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.713+0000 7f73f77fe640 1 -- 172.21.15.114:0/4281504079 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f73f81023d0 msgr2=0x7f73f8101cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:45.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.713+0000 7f73f77fe640 1 --2- 172.21.15.114:0/4281504079 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f73f81023d0 0x7f73f8101cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:45.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.713+0000 7f73f77fe640 1 -- 172.21.15.114:0/4281504079 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f81008c0 con 0x7f73f8104d00 2024-09-20T19:44:45.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.713+0000 7f73f77fe640 1 --2- 172.21.15.114:0/4281504079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f8104d00 0x7f73f8100380 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f73e000d6c0 tx=0x7f73e000db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:45.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.713+0000 7f73f57fa640 1 -- 172.21.15.114:0/4281504079 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f73e0017070 con 0x7f73f8104d00 2024-09-20T19:44:45.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.713+0000 7f73fe907640 1 -- 172.21.15.114:0/4281504079 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f8100bb0 con 0x7f73f8104d00 2024-09-20T19:44:45.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.714+0000 7f73fe907640 1 -- 172.21.15.114:0/4281504079 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f8101140 con 0x7f73f8104d00 2024-09-20T19:44:45.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.714+0000 7f73f57fa640 1 -- 172.21.15.114:0/4281504079 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f73e00054e0 con 0x7f73f8104d00 2024-09-20T19:44:45.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.714+0000 7f73f57fa640 1 -- 172.21.15.114:0/4281504079 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f73e0004b70 con 0x7f73f8104d00 2024-09-20T19:44:45.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.715+0000 7f73fe907640 1 -- 172.21.15.114:0/4281504079 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73d80052c0 con 0x7f73f8104d00 2024-09-20T19:44:45.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.716+0000 7f73f57fa640 1 -- 172.21.15.114:0/4281504079 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f73e0004de0 con 0x7f73f8104d00 2024-09-20T19:44:45.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.716+0000 7f73f57fa640 1 --2- 172.21.15.114:0/4281504079 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f73e406e300 0x7f73e40707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:45.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.717+0000 7f73f57fa640 1 -- 172.21.15.114:0/4281504079 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f73e008df60 con 0x7f73f8104d00 2024-09-20T19:44:45.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.717+0000 7f73f7fff640 1 --2- 172.21.15.114:0/4281504079 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f73e406e300 0x7f73e40707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:45.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.717+0000 7f73f7fff640 1 --2- 172.21.15.114:0/4281504079 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f73e406e300 0x7f73e40707c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f73e8006010 tx=0x7f73e8005a20 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:45.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:45.720+0000 7f73f57fa640 1 -- 172.21.15.114:0/4281504079 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f73e00bd210 con 0x7f73f8104d00 2024-09-20T19:44:45.735 INFO:tasks.workunit.client.1.smithi160.stdout:8/101: sync 2024-09-20T19:44:45.736 INFO:tasks.workunit.client.1.smithi160.stdout:4/156: sync 2024-09-20T19:44:45.736 INFO:tasks.workunit.client.1.smithi160.stdout:0/69: sync 2024-09-20T19:44:45.736 INFO:tasks.workunit.client.1.smithi160.stdout:4/157: fdatasync d0/d10/d14/d24/f27 0 2024-09-20T19:44:45.737 INFO:tasks.workunit.client.1.smithi160.stdout:8/102: unlink d1/c11 0 2024-09-20T19:44:45.737 INFO:tasks.workunit.client.1.smithi160.stdout:8/103: chown d1/d14/f1d 2144204 1 2024-09-20T19:44:45.738 INFO:tasks.workunit.client.1.smithi160.stdout:8/104: truncate d1/d14/f1d 368727 0 2024-09-20T19:44:45.742 INFO:tasks.workunit.client.1.smithi160.stdout:4/158: read d0/d10/d14/d24/f27 [601190,77779] 0 2024-09-20T19:44:45.742 INFO:tasks.workunit.client.1.smithi160.stdout:4/159: getdents d0/d19/d32 0 2024-09-20T19:44:45.743 INFO:tasks.workunit.client.1.smithi160.stdout:4/160: fsync d0/d10/d14/d24/d9/f17 0 2024-09-20T19:44:45.751 INFO:tasks.workunit.client.1.smithi160.stdout:6/126: dread d8/d9/f1b [0,4194304] 0 2024-09-20T19:44:45.754 INFO:tasks.workunit.client.1.smithi160.stdout:6/127: mkdir d8/d27 0 2024-09-20T19:44:45.757 INFO:tasks.workunit.client.1.smithi160.stdout:9/148: dwrite d0/f1e [0,4194304] 0 2024-09-20T19:44:45.758 INFO:tasks.workunit.client.1.smithi160.stdout:9/149: dread - d0/d4/d5/f2a zero size 2024-09-20T19:44:45.758 INFO:tasks.workunit.client.1.smithi160.stdout:9/150: readlink d0/l1d 0 2024-09-20T19:44:45.760 INFO:tasks.workunit.client.1.smithi160.stdout:2/180: dwrite d7/d10/d29/f30 [0,4194304] 0 2024-09-20T19:44:45.761 INFO:tasks.workunit.client.1.smithi160.stdout:6/128: rename d8/l14 to d8/d27/l28 0 2024-09-20T19:44:45.766 INFO:tasks.workunit.client.1.smithi160.stdout:6/129: read d8/d9/f1b [3055349,39961] 0 2024-09-20T19:44:45.768 INFO:tasks.workunit.client.1.smithi160.stdout:2/181: mkdir d7/d10/d29/d31/d3c 0 2024-09-20T19:44:45.769 INFO:tasks.workunit.client.1.smithi160.stdout:9/151: dread d0/d4/f1b [0,4194304] 0 2024-09-20T19:44:45.771 INFO:tasks.workunit.client.1.smithi160.stdout:5/117: dwrite d1/d7/da/d20/f21 [0,4194304] 0 2024-09-20T19:44:45.772 INFO:tasks.workunit.client.1.smithi160.stdout:6/130: creat d8/d27/f29 x:0 0 0 2024-09-20T19:44:45.773 INFO:tasks.workunit.client.1.smithi160.stdout:5/118: dread d1/d7/ff [0,4194304] 0 2024-09-20T19:44:45.773 INFO:tasks.workunit.client.1.smithi160.stdout:9/152: creat d0/f30 x:0 0 0 2024-09-20T19:44:45.774 INFO:tasks.workunit.client.1.smithi160.stdout:9/153: write d0/d4/f1a [787200,42960] 0 2024-09-20T19:44:45.775 INFO:tasks.workunit.client.1.smithi160.stdout:0/70: dwrite d4/d6/fa [4194304,4194304] 0 2024-09-20T19:44:45.775 INFO:tasks.workunit.client.1.smithi160.stdout:0/71: chown d4/d6/df 344 1 2024-09-20T19:44:45.775 INFO:tasks.workunit.client.1.smithi160.stdout:0/72: dread - d4/f15 zero size 2024-09-20T19:44:45.776 INFO:tasks.workunit.client.1.smithi160.stdout:5/119: symlink d1/d7/l2a 0 2024-09-20T19:44:45.777 INFO:tasks.workunit.client.1.smithi160.stdout:9/154: symlink d0/d4/d5/l31 0 2024-09-20T19:44:45.778 INFO:tasks.workunit.client.1.smithi160.stdout:4/161: dwrite d0/d10/f18 [0,4194304] 0 2024-09-20T19:44:45.778 INFO:tasks.workunit.client.1.smithi160.stdout:1/95: fsync d6/da/fb 0 2024-09-20T19:44:45.779 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Sep 20 19:44:45 smithi160 ceph-mon[35859]: pgmap v238: 65 pgs: 65 active+clean; 195 MiB data, 898 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 9.8 MiB/s wr, 892 op/s 2024-09-20T19:44:45.780 INFO:tasks.workunit.client.1.smithi160.stdout:7/87: dwrite f0 [0,4194304] 0 2024-09-20T19:44:45.780 INFO:tasks.workunit.client.1.smithi160.stdout:7/88: write f0 [4710499,16803] 0 2024-09-20T19:44:45.780 INFO:tasks.workunit.client.1.smithi160.stdout:2/182: dread d7/d10/d29/d24/f2f [0,4194304] 0 2024-09-20T19:44:45.780 INFO:tasks.workunit.client.1.smithi160.stdout:8/105: dwrite d1/df/d13/f1a [0,4194304] 0 2024-09-20T19:44:45.781 INFO:tasks.workunit.client.1.smithi160.stdout:2/183: write d7/f17 [1274090,80459] 0 2024-09-20T19:44:45.781 INFO:tasks.workunit.client.1.smithi160.stdout:2/184: fdatasync d7/d10/f1c 0 2024-09-20T19:44:45.781 INFO:tasks.workunit.client.1.smithi160.stdout:0/73: creat d4/d6/db/f17 x:0 0 0 2024-09-20T19:44:45.787 INFO:tasks.workunit.client.1.smithi160.stdout:5/120: symlink d1/l2b 0 2024-09-20T19:44:45.790 INFO:tasks.workunit.client.1.smithi160.stdout:9/155: rename d0/d8/f28 to d0/d4/f32 0 2024-09-20T19:44:45.790 INFO:tasks.workunit.client.1.smithi160.stdout:9/156: truncate d0/f30 475180 0 2024-09-20T19:44:45.790 INFO:tasks.workunit.client.1.smithi160.stdout:6/131: dwrite d8/f21 [0,4194304] 0 2024-09-20T19:44:45.793 INFO:tasks.workunit.client.1.smithi160.stdout:2/185: write d7/d10/d29/f30 [2373029,88500] 0 2024-09-20T19:44:45.794 INFO:tasks.workunit.client.1.smithi160.stdout:4/162: mknod d0/d10/c35 0 2024-09-20T19:44:45.801 INFO:tasks.workunit.client.1.smithi160.stdout:9/157: dread d0/fc [0,4194304] 0 2024-09-20T19:44:45.802 INFO:tasks.workunit.client.1.smithi160.stdout:1/96: mkdir d6/da/d13/d1a 0 2024-09-20T19:44:45.802 INFO:tasks.workunit.client.1.smithi160.stdout:1/97: chown l1 1044981 1 2024-09-20T19:44:45.812 INFO:tasks.workunit.client.1.smithi160.stdout:4/163: dwrite d0/d31/f15 [4194304,4194304] 0 2024-09-20T19:44:45.812 INFO:tasks.workunit.client.1.smithi160.stdout:7/89: truncate d3/d4/f16 3140621 0 2024-09-20T19:44:45.817 INFO:tasks.workunit.client.1.smithi160.stdout:5/121: creat d1/d7/f2c x:0 0 0 2024-09-20T19:44:45.817 INFO:tasks.workunit.client.1.smithi160.stdout:5/122: chown d1/d7/da/f16 19 1 2024-09-20T19:44:45.817 INFO:tasks.workunit.client.1.smithi160.stdout:5/123: chown d1/f5 40998313 1 2024-09-20T19:44:45.817 INFO:tasks.workunit.client.1.smithi160.stdout:5/124: fsync d1/d7/da/f16 0 2024-09-20T19:44:45.820 INFO:tasks.workunit.client.1.smithi160.stdout:8/106: rename d1/fb to d1/df/d13/d1b/f21 0 2024-09-20T19:44:45.826 INFO:tasks.workunit.client.1.smithi160.stdout:9/158: mknod d0/d4/c33 0 2024-09-20T19:44:45.826 INFO:tasks.workunit.client.1.smithi160.stdout:9/159: chown d0/d8/d16/d17/c2b 1873609559 1 2024-09-20T19:44:46.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.070+0000 7f73fe907640 1 -- 172.21.15.114:0/4281504079 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7f73d8000bf0 con 0x7f73e406e300 2024-09-20T19:44:46.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.071+0000 7f73f57fa640 1 -- 172.21.15.114:0/4281504079 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+439 (secure 0 0 0) 0x7f73d8000bf0 con 0x7f73e406e300 2024-09-20T19:44:46.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.074+0000 7f73fe907640 1 -- 172.21.15.114:0/4281504079 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f73e406e300 msgr2=0x7f73e40707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.074+0000 7f73fe907640 1 --2- 172.21.15.114:0/4281504079 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f73e406e300 0x7f73e40707c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f73e8006010 tx=0x7f73e8005a20 comp rx=0 tx=0).stop 2024-09-20T19:44:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.074+0000 7f73fe907640 1 -- 172.21.15.114:0/4281504079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f8104d00 msgr2=0x7f73f8100380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.074+0000 7f73fe907640 1 --2- 172.21.15.114:0/4281504079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f8104d00 0x7f73f8100380 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f73e000d6c0 tx=0x7f73e000db80 comp rx=0 tx=0).stop 2024-09-20T19:44:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.074+0000 7f73fe907640 1 -- 172.21.15.114:0/4281504079 shutdown_connections 2024-09-20T19:44:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.074+0000 7f73fe907640 1 --2- 172.21.15.114:0/4281504079 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f73e406e300 0x7f73e40707c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.074+0000 7f73fe907640 1 --2- 172.21.15.114:0/4281504079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f8104d00 0x7f73f8100380 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.074+0000 7f73fe907640 1 --2- 172.21.15.114:0/4281504079 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f73f81023d0 0x7f73f8101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.074+0000 7f73fe907640 1 -- 172.21.15.114:0/4281504079 >> 172.21.15.114:0/4281504079 conn(0x7f73f80fbff0 msgr2=0x7f73f80fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:46.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.075+0000 7f73fe907640 1 -- 172.21.15.114:0/4281504079 shutdown_connections 2024-09-20T19:44:46.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.075+0000 7f73fe907640 1 -- 172.21.15.114:0/4281504079 wait complete. 2024-09-20T19:44:46.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.173+0000 7fcb71eaf640 1 -- 172.21.15.114:0/3580273608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6c102430 msgr2=0x7fcb6c104820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:46.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.173+0000 7fcb71eaf640 1 --2- 172.21.15.114:0/3580273608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6c102430 0x7fcb6c104820 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009880 tx=0x7fcb5c02a4d0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.173+0000 7fcb71eaf640 1 -- 172.21.15.114:0/3580273608 shutdown_connections 2024-09-20T19:44:46.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.173+0000 7fcb71eaf640 1 --2- 172.21.15.114:0/3580273608 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcb6c104d60 0x7fcb6c107150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.173+0000 7fcb71eaf640 1 --2- 172.21.15.114:0/3580273608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6c102430 0x7fcb6c104820 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.173+0000 7fcb71eaf640 1 -- 172.21.15.114:0/3580273608 >> 172.21.15.114:0/3580273608 conn(0x7fcb6c0fbff0 msgr2=0x7fcb6c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:46.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.173+0000 7fcb71eaf640 1 -- 172.21.15.114:0/3580273608 shutdown_connections 2024-09-20T19:44:46.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.173+0000 7fcb71eaf640 1 -- 172.21.15.114:0/3580273608 wait complete. 2024-09-20T19:44:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.174+0000 7fcb71eaf640 1 Processor -- start 2024-09-20T19:44:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.174+0000 7fcb71eaf640 1 -- start start 2024-09-20T19:44:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.174+0000 7fcb71eaf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6c102430 0x7fcb6c1974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.174+0000 7fcb71eaf640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcb6c104d60 0x7fcb6c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.174+0000 7fcb71eaf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c10a7a0 con 0x7fcb6c102430 2024-09-20T19:44:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.174+0000 7fcb71eaf640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c10a620 con 0x7fcb6c104d60 2024-09-20T19:44:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.175+0000 7fcb6b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6c102430 0x7fcb6c1974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.175+0000 7fcb6b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6c102430 0x7fcb6c1974c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49324/0 (socket says 172.21.15.114:49324) 2024-09-20T19:44:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.175+0000 7fcb6b7fe640 1 -- 172.21.15.114:0/3833546633 learned_addr learned my addr 172.21.15.114:0/3833546633 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:44:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.175+0000 7fcb6b7fe640 1 -- 172.21.15.114:0/3833546633 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcb6c104d60 msgr2=0x7fcb6c197a00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-20T19:44:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.175+0000 7fcb6b7fe640 1 --2- 172.21.15.114:0/3833546633 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcb6c104d60 0x7fcb6c197a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.175+0000 7fcb6b7fe640 1 -- 172.21.15.114:0/3833546633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6c198080 con 0x7fcb6c102430 2024-09-20T19:44:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.175+0000 7fcb6b7fe640 1 --2- 172.21.15.114:0/3833546633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6c102430 0x7fcb6c1974c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c006010 tx=0x7fcb5c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.175+0000 7fcb68ff9640 1 -- 172.21.15.114:0/3833546633 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fcb5c03f070 con 0x7fcb6c102430 2024-09-20T19:44:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.175+0000 7fcb71eaf640 1 -- 172.21.15.114:0/3833546633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6c100280 con 0x7fcb6c102430 2024-09-20T19:44:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.175+0000 7fcb68ff9640 1 -- 172.21.15.114:0/3833546633 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcb5c0044d0 con 0x7fcb6c102430 2024-09-20T19:44:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.175+0000 7fcb68ff9640 1 -- 172.21.15.114:0/3833546633 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fcb5c035030 con 0x7fcb6c102430 2024-09-20T19:44:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.175+0000 7fcb71eaf640 1 -- 172.21.15.114:0/3833546633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb6c100890 con 0x7fcb6c102430 2024-09-20T19:44:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.176+0000 7fcb71eaf640 1 -- 172.21.15.114:0/3833546633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb6c102810 con 0x7fcb6c102430 2024-09-20T19:44:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.177+0000 7fcb68ff9640 1 -- 172.21.15.114:0/3833546633 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7fcb5c042070 con 0x7fcb6c102430 2024-09-20T19:44:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.177+0000 7fcb68ff9640 1 --2- 172.21.15.114:0/3833546633 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fcb5406e310 0x7fcb540707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:46.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.177+0000 7fcb68ff9640 1 -- 172.21.15.114:0/3833546633 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7fcb5c0aeda0 con 0x7fcb6c102430 2024-09-20T19:44:46.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.178+0000 7fcb6affd640 1 --2- 172.21.15.114:0/3833546633 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fcb5406e310 0x7fcb540707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:46.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.178+0000 7fcb6affd640 1 --2- 172.21.15.114:0/3833546633 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fcb5406e310 0x7fcb540707d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c198970 tx=0x7fcb60006bd0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:46.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.182+0000 7fcb68ff9640 1 -- 172.21.15.114:0/3833546633 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fcb5c07c780 con 0x7fcb6c102430 2024-09-20T19:44:46.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.408+0000 7fcb71eaf640 1 -- 172.21.15.114:0/3833546633 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fcb6c1057c0 con 0x7fcb5406e310 2024-09-20T19:44:46.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.420+0000 7fcb68ff9640 1 -- 172.21.15.114:0/3833546633 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3768 (secure 0 0 0) 0x7fcb6c1057c0 con 0x7fcb5406e310 2024-09-20T19:44:46.421 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T19:44:46.421 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (5m) 107s ago 7m 26.3M - 0.25.0 c8568f914cd2 d19ec799e98c 2024-09-20T19:44:46.422 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (7m) 107s ago 7m 8220k - 19.1.1-369-ged8ee0f2 6acb0a031eab ddd91f706b5f 2024-09-20T19:44:46.422 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi160 smithi160 running (6m) 111s ago 6m 5268k - 19.1.1-369-ged8ee0f2 6acb0a031eab 8ce900c6456f 2024-09-20T19:44:46.422 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (7m) 107s ago 7m 6895k - 19.1.1-369-ged8ee0f2 6acb0a031eab 0e32ab0eeecd 2024-09-20T19:44:46.422 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi160 smithi160 running (6m) 111s ago 6m 6891k - 19.1.1-369-ged8ee0f2 6acb0a031eab d858e40463d6 2024-09-20T19:44:46.422 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (5m) 107s ago 7m 78.8M - 9.4.12 2bacad6d85d8 cd034302ef4b 2024-09-20T19:44:46.422 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.scsmgq smithi114 running (2m) 107s ago 2m 16.3M - 19.1.1-369-ged8ee0f2 6acb0a031eab fcac0f36c17d 2024-09-20T19:44:46.422 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tlrcnf smithi114 running (2m) 107s ago 2m 16.0M - 19.1.1-369-ged8ee0f2 6acb0a031eab db741608c3d0 2024-09-20T19:44:46.422 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi160.lylyja smithi160 running (2m) 111s ago 2m 14.6M - 19.1.1-369-ged8ee0f2 6acb0a031eab aa5d71040f0c 2024-09-20T19:44:46.422 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi160.xirihp smithi160 running (2m) 111s ago 2m 15.4M - 19.1.1-369-ged8ee0f2 6acb0a031eab da5866e71f3a 2024-09-20T19:44:46.422 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.rvhgji smithi114 *:9283,8765,8443 running (8m) 107s ago 8m 498M - 19.1.1-369-ged8ee0f2 6acb0a031eab 59997c1e986a 2024-09-20T19:44:46.422 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi160.bspiak smithi160 *:8443,9283,8765 running (6m) 111s ago 6m 441M - 19.1.1-369-ged8ee0f2 6acb0a031eab 50306543ba66 2024-09-20T19:44:46.422 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (8m) 107s ago 8m 47.9M 2048M 19.1.1-369-ged8ee0f2 6acb0a031eab ed7f013f2513 2024-09-20T19:44:46.422 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi160 smithi160 running (6m) 111s ago 6m 34.6M 2048M 19.1.1-369-ged8ee0f2 6acb0a031eab 961cc10b05cc 2024-09-20T19:44:46.423 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (7m) 107s ago 7m 16.3M - 1.5.0 0da6a335fe13 6542a55ff841 2024-09-20T19:44:46.423 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi160 smithi160 *:9100 running (6m) 111s ago 6m 16.4M - 1.5.0 0da6a335fe13 ff98c99cbe97 2024-09-20T19:44:46.423 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (5m) 107s ago 5m 47.9M 3946M 19.1.1-369-ged8ee0f2 6acb0a031eab 6f0510eaf505 2024-09-20T19:44:46.423 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (4m) 107s ago 4m 53.8M 3946M 19.1.1-369-ged8ee0f2 6acb0a031eab 98b6e5128717 2024-09-20T19:44:46.423 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (4m) 107s ago 4m 46.5M 3946M 19.1.1-369-ged8ee0f2 6acb0a031eab dda945a7e7ca 2024-09-20T19:44:46.423 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi160 running (4m) 111s ago 4m 52.7M 4970M 19.1.1-369-ged8ee0f2 6acb0a031eab 9facdfb1a166 2024-09-20T19:44:46.423 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi160 running (3m) 111s ago 3m 45.3M 4970M 19.1.1-369-ged8ee0f2 6acb0a031eab e15086d3d575 2024-09-20T19:44:46.423 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi160 running (3m) 111s ago 3m 51.3M 4970M 19.1.1-369-ged8ee0f2 6acb0a031eab 7f90f136e99a 2024-09-20T19:44:46.423 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (5m) 107s ago 6m 60.2M - 2.43.0 a07b618ecd1d 733ef913ac0b 2024-09-20T19:44:46.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.422+0000 7fcb71eaf640 1 -- 172.21.15.114:0/3833546633 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fcb5406e310 msgr2=0x7fcb540707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:46.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.422+0000 7fcb71eaf640 1 --2- 172.21.15.114:0/3833546633 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fcb5406e310 0x7fcb540707d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c198970 tx=0x7fcb60006bd0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.422+0000 7fcb71eaf640 1 -- 172.21.15.114:0/3833546633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6c102430 msgr2=0x7fcb6c1974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:46.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.422+0000 7fcb71eaf640 1 --2- 172.21.15.114:0/3833546633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6c102430 0x7fcb6c1974c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c006010 tx=0x7fcb5c0048f0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.422+0000 7fcb71eaf640 1 -- 172.21.15.114:0/3833546633 shutdown_connections 2024-09-20T19:44:46.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.422+0000 7fcb71eaf640 1 --2- 172.21.15.114:0/3833546633 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7fcb5406e310 0x7fcb540707d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.422+0000 7fcb71eaf640 1 --2- 172.21.15.114:0/3833546633 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcb6c104d60 0x7fcb6c197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.422+0000 7fcb71eaf640 1 --2- 172.21.15.114:0/3833546633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb6c102430 0x7fcb6c1974c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.422+0000 7fcb71eaf640 1 -- 172.21.15.114:0/3833546633 >> 172.21.15.114:0/3833546633 conn(0x7fcb6c0fbff0 msgr2=0x7fcb6c0fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:46.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.422+0000 7fcb71eaf640 1 -- 172.21.15.114:0/3833546633 shutdown_connections 2024-09-20T19:44:46.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.422+0000 7fcb71eaf640 1 -- 172.21.15.114:0/3833546633 wait complete. 2024-09-20T19:44:46.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.503+0000 7f84a2ccb640 1 -- 172.21.15.114:0/1930616471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c104d60 msgr2=0x7f849c107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:46.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.503+0000 7f84a2ccb640 1 --2- 172.21.15.114:0/1930616471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c104d60 0x7f849c107150 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f848c009880 tx=0x7f848c02a4d0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.504+0000 7f84a2ccb640 1 -- 172.21.15.114:0/1930616471 shutdown_connections 2024-09-20T19:44:46.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.504+0000 7f84a2ccb640 1 --2- 172.21.15.114:0/1930616471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c104d60 0x7f849c107150 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.504+0000 7f84a2ccb640 1 --2- 172.21.15.114:0/1930616471 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f849c102430 0x7f849c104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.504+0000 7f84a2ccb640 1 -- 172.21.15.114:0/1930616471 >> 172.21.15.114:0/1930616471 conn(0x7f849c0fbff0 msgr2=0x7f849c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:46.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.505+0000 7f84a2ccb640 1 -- 172.21.15.114:0/1930616471 shutdown_connections 2024-09-20T19:44:46.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.505+0000 7f84a2ccb640 1 -- 172.21.15.114:0/1930616471 wait complete. 2024-09-20T19:44:46.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.505+0000 7f84a2ccb640 1 Processor -- start 2024-09-20T19:44:46.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.505+0000 7f84a2ccb640 1 -- start start 2024-09-20T19:44:46.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.505+0000 7f84a2ccb640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f849c102430 0x7f849c197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:46.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.505+0000 7f84a2ccb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c104d60 0x7f849c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:46.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.505+0000 7f84a2ccb640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f849c10a7a0 con 0x7f849c104d60 2024-09-20T19:44:46.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.505+0000 7f84a2ccb640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f849c10a620 con 0x7f849c102430 2024-09-20T19:44:46.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.505+0000 7f8493fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c104d60 0x7f849c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:46.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.506+0000 7f8493fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c104d60 0x7f849c197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49338/0 (socket says 172.21.15.114:49338) 2024-09-20T19:44:46.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.506+0000 7f8493fff640 1 -- 172.21.15.114:0/153246775 learned_addr learned my addr 172.21.15.114:0/153246775 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:44:46.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.506+0000 7f8493fff640 1 -- 172.21.15.114:0/153246775 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f849c102430 msgr2=0x7f849c197500 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-20T19:44:46.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.506+0000 7f8493fff640 1 --2- 172.21.15.114:0/153246775 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f849c102430 0x7f849c197500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.506+0000 7f8493fff640 1 -- 172.21.15.114:0/153246775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849c1980c0 con 0x7f849c104d60 2024-09-20T19:44:46.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.506+0000 7f8493fff640 1 --2- 172.21.15.114:0/153246775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c104d60 0x7f849c197a40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f848c006010 tx=0x7f848c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:46.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.506+0000 7f8491ffb640 1 -- 172.21.15.114:0/153246775 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f848c03f070 con 0x7f849c104d60 2024-09-20T19:44:46.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.506+0000 7f84a2ccb640 1 -- 172.21.15.114:0/153246775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f849c19cb00 con 0x7f849c104d60 2024-09-20T19:44:46.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.506+0000 7f8491ffb640 1 -- 172.21.15.114:0/153246775 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f848c0044d0 con 0x7f849c104d60 2024-09-20T19:44:46.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.506+0000 7f8491ffb640 1 -- 172.21.15.114:0/153246775 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f848c035030 con 0x7f849c104d60 2024-09-20T19:44:46.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.507+0000 7f84a2ccb640 1 -- 172.21.15.114:0/153246775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f849c19d110 con 0x7f849c104d60 2024-09-20T19:44:46.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.507+0000 7f84a2ccb640 1 -- 172.21.15.114:0/153246775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f849c102810 con 0x7f849c104d60 2024-09-20T19:44:46.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.508+0000 7f8491ffb640 1 -- 172.21.15.114:0/153246775 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f848c042070 con 0x7f849c104d60 2024-09-20T19:44:46.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.509+0000 7f8491ffb640 1 --2- 172.21.15.114:0/153246775 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f848406e2b0 0x7f8484070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:46.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.509+0000 7f8491ffb640 1 -- 172.21.15.114:0/153246775 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f848c0aed80 con 0x7f849c104d60 2024-09-20T19:44:46.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.509+0000 7f84a0a67640 1 --2- 172.21.15.114:0/153246775 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f848406e2b0 0x7f8484070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:46.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.509+0000 7f84a0a67640 1 --2- 172.21.15.114:0/153246775 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f848406e2b0 0x7f8484070770 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8498009670 tx=0x7f8498006bd0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:46.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.514+0000 7f8491ffb640 1 -- 172.21.15.114:0/153246775 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f848c07c6e0 con 0x7f849c104d60 2024-09-20T19:44:46.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.800+0000 7f84a2ccb640 1 -- 172.21.15.114:0/153246775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) -- 0x7f849c19d440 con 0x7f849c104d60 2024-09-20T19:44:46.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.805+0000 7f8491ffb640 1 -- 172.21.15.114:0/153246775 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) ==== 56+0+633 (secure 0 0 0) 0x7f848c080390 con 0x7f849c104d60 2024-09-20T19:44:46.807 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-20T19:44:46.807 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-09-20T19:44:46.807 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)": 2 2024-09-20T19:44:46.807 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-20T19:44:46.807 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-09-20T19:44:46.807 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)": 2 2024-09-20T19:44:46.807 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-20T19:44:46.807 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-09-20T19:44:46.807 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)": 6 2024-09-20T19:44:46.807 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-20T19:44:46.807 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-09-20T19:44:46.807 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)": 4 2024-09-20T19:44:46.807 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-20T19:44:46.808 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-09-20T19:44:46.808 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.1.1-369-ged8ee0f2 (ed8ee0f26ef720e6504cefa8be1a2d0872192de7) squid (stable)": 14 2024-09-20T19:44:46.808 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-20T19:44:46.808 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-20T19:44:46.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.807+0000 7f84a2ccb640 1 -- 172.21.15.114:0/153246775 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f848406e2b0 msgr2=0x7f8484070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:46.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.807+0000 7f84a2ccb640 1 --2- 172.21.15.114:0/153246775 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f848406e2b0 0x7f8484070770 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8498009670 tx=0x7f8498006bd0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.807+0000 7f84a2ccb640 1 -- 172.21.15.114:0/153246775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c104d60 msgr2=0x7f849c197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:46.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.807+0000 7f84a2ccb640 1 --2- 172.21.15.114:0/153246775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c104d60 0x7f849c197a40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f848c006010 tx=0x7f848c0048f0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.807+0000 7f84a2ccb640 1 -- 172.21.15.114:0/153246775 shutdown_connections 2024-09-20T19:44:46.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.807+0000 7f84a2ccb640 1 --2- 172.21.15.114:0/153246775 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f848406e2b0 0x7f8484070770 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.807+0000 7f84a2ccb640 1 --2- 172.21.15.114:0/153246775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c104d60 0x7f849c197a40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.807+0000 7f84a2ccb640 1 --2- 172.21.15.114:0/153246775 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f849c102430 0x7f849c197500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.807+0000 7f84a2ccb640 1 -- 172.21.15.114:0/153246775 >> 172.21.15.114:0/153246775 conn(0x7f849c0fbff0 msgr2=0x7f849c0fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:46.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.807+0000 7f84a2ccb640 1 -- 172.21.15.114:0/153246775 shutdown_connections 2024-09-20T19:44:46.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.808+0000 7f84a2ccb640 1 -- 172.21.15.114:0/153246775 wait complete. 2024-09-20T19:44:46.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.891+0000 7f14c08c1640 1 -- 172.21.15.114:0/3725056306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14bc100840 msgr2=0x7f14bc100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.891+0000 7f14c08c1640 1 --2- 172.21.15.114:0/3725056306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14bc100840 0x7f14bc100c40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009880 tx=0x7f14ac02a4d0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.892+0000 7f14c08c1640 1 -- 172.21.15.114:0/3725056306 shutdown_connections 2024-09-20T19:44:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.892+0000 7f14c08c1640 1 --2- 172.21.15.114:0/3725056306 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f14bc101a40 0x7f14bc101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.892+0000 7f14c08c1640 1 --2- 172.21.15.114:0/3725056306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14bc100840 0x7f14bc100c40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.892+0000 7f14c08c1640 1 -- 172.21.15.114:0/3725056306 >> 172.21.15.114:0/3725056306 conn(0x7f14bc0fbff0 msgr2=0x7f14bc0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.892+0000 7f14c08c1640 1 -- 172.21.15.114:0/3725056306 shutdown_connections 2024-09-20T19:44:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.892+0000 7f14c08c1640 1 -- 172.21.15.114:0/3725056306 wait complete. 2024-09-20T19:44:46.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.892+0000 7f14c08c1640 1 Processor -- start 2024-09-20T19:44:46.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.892+0000 7f14c08c1640 1 -- start start 2024-09-20T19:44:46.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14c08c1640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f14bc100840 0x7f14bc197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:46.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14c08c1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14bc101a40 0x7f14bc197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:46.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14c08c1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc109700 con 0x7f14bc101a40 2024-09-20T19:44:46.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14c08c1640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc109580 con 0x7f14bc100840 2024-09-20T19:44:46.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14b9d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14bc101a40 0x7f14bc197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:46.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14b9d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14bc101a40 0x7f14bc197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49352/0 (socket says 172.21.15.114:49352) 2024-09-20T19:44:46.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14b9d9b640 1 -- 172.21.15.114:0/2196777676 learned_addr learned my addr 172.21.15.114:0/2196777676 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-20T19:44:46.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14b9d9b640 1 -- 172.21.15.114:0/2196777676 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f14bc100840 msgr2=0x7f14bc197580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-20T19:44:46.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14b9d9b640 1 --2- 172.21.15.114:0/2196777676 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f14bc100840 0x7f14bc197580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:46.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14b9d9b640 1 -- 172.21.15.114:0/2196777676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14bc198140 con 0x7f14bc101a40 2024-09-20T19:44:46.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14b9d9b640 1 --2- 172.21.15.114:0/2196777676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14bc101a40 0x7f14bc197ac0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f14b000b3d0 tx=0x7f14b000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:46.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14ab7fe640 1 -- 172.21.15.114:0/2196777676 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f14b0015020 con 0x7f14bc101a40 2024-09-20T19:44:46.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14c08c1640 1 -- 172.21.15.114:0/2196777676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14bc19cb80 con 0x7f14bc101a40 2024-09-20T19:44:46.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14ab7fe640 1 -- 172.21.15.114:0/2196777676 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f14b0011af0 con 0x7f14bc101a40 2024-09-20T19:44:46.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14ab7fe640 1 -- 172.21.15.114:0/2196777676 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f14b0010da0 con 0x7f14bc101a40 2024-09-20T19:44:46.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.893+0000 7f14c08c1640 1 -- 172.21.15.114:0/2196777676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14bc19d110 con 0x7f14bc101a40 2024-09-20T19:44:46.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.895+0000 7f14c08c1640 1 -- 172.21.15.114:0/2196777676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14bc100c40 con 0x7f14bc101a40 2024-09-20T19:44:46.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.896+0000 7f14ab7fe640 1 -- 172.21.15.114:0/2196777676 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 92553+0+0 (secure 0 0 0) 0x7f14b0018020 con 0x7f14bc101a40 2024-09-20T19:44:46.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.896+0000 7f14ab7fe640 1 --2- 172.21.15.114:0/2196777676 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f14a406e310 0x7f14a40707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-20T19:44:46.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.896+0000 7f14ab7fe640 1 -- 172.21.15.114:0/2196777676 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5468+0+0 (secure 0 0 0) 0x7f14b008dc80 con 0x7f14bc101a40 2024-09-20T19:44:46.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.896+0000 7f14ba59c640 1 --2- 172.21.15.114:0/2196777676 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f14a406e310 0x7f14a40707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-20T19:44:46.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.897+0000 7f14ba59c640 1 --2- 172.21.15.114:0/2196777676 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f14a406e310 0x7f14a40707d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f14ac02a9f0 tx=0x7f14ac000bc0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-20T19:44:46.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:46.901+0000 7f14ab7fe640 1 -- 172.21.15.114:0/2196777676 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f14b001f020 con 0x7f14bc101a40 2024-09-20T19:44:47.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:47.141+0000 7f14c08c1640 1 -- 172.21.15.114:0/2196777676 --> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7f14bc06a2f0 con 0x7f14a406e310 2024-09-20T19:44:47.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:47.143+0000 7f14ab7fe640 1 -- 172.21.15.114:0/2196777676 <== mgr.14219 v2:172.21.15.114:6800/1083342253 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+439 (secure 0 0 0) 0x7f14bc06a2f0 con 0x7f14a406e310 2024-09-20T19:44:47.144 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-20T19:44:47.144 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T19:44:47.145 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-09-20T19:44:47.145 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-20T19:44:47.145 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [], 2024-09-20T19:44:47.145 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "0/2 daemons upgraded", 2024-09-20T19:44:47.145 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d image on host smithi160", 2024-09-20T19:44:47.145 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-09-20T19:44:47.145 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-20T19:44:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:47.145+0000 7f14c08c1640 1 -- 172.21.15.114:0/2196777676 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f14a406e310 msgr2=0x7f14a40707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:47.145+0000 7f14c08c1640 1 --2- 172.21.15.114:0/2196777676 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f14a406e310 0x7f14a40707d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f14ac02a9f0 tx=0x7f14ac000bc0 comp rx=0 tx=0).stop 2024-09-20T19:44:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:47.145+0000 7f14c08c1640 1 -- 172.21.15.114:0/2196777676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14bc101a40 msgr2=0x7f14bc197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-20T19:44:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:47.145+0000 7f14c08c1640 1 --2- 172.21.15.114:0/2196777676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14bc101a40 0x7f14bc197ac0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f14b000b3d0 tx=0x7f14b000b890 comp rx=0 tx=0).stop 2024-09-20T19:44:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:47.145+0000 7f14c08c1640 1 -- 172.21.15.114:0/2196777676 shutdown_connections 2024-09-20T19:44:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:47.145+0000 7f14c08c1640 1 --2- 172.21.15.114:0/2196777676 >> [v2:172.21.15.114:6800/1083342253,v1:172.21.15.114:6801/1083342253] conn(0x7f14a406e310 0x7f14a40707d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:47.145+0000 7f14c08c1640 1 --2- 172.21.15.114:0/2196777676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14bc101a40 0x7f14bc197ac0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:47.145+0000 7f14c08c1640 1 --2- 172.21.15.114:0/2196777676 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f14bc100840 0x7f14bc197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-20T19:44:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:47.145+0000 7f14c08c1640 1 -- 172.21.15.114:0/2196777676 >> 172.21.15.114:0/2196777676 conn(0x7f14bc0fbff0 msgr2=0x7f14bc069be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-20T19:44:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:47.145+0000 7f14c08c1640 1 -- 172.21.15.114:0/2196777676 shutdown_connections 2024-09-20T19:44:47.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:44:47.145+0000 7f14c08c1640 1 -- 172.21.15.114:0/2196777676 wait complete. 2024-09-20T19:45:07.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:06.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:07.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:06.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:07.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:06.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:06.911+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:06.911+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:06.911+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:08.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:07.878+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:08.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:07.878+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:08.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:07.878+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:07.864+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:07.864+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:07.864+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:08.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:08.878+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:08.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:08.878+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:08.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:08.878+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:08.884+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:08.884+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:08.884+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:09.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:09.082+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:09.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:09.082+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:09.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:09.082+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:10.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:09.911+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:10.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:09.911+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:10.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:09.911+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:10.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:10.050+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:10.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:10.050+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:10.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:10.050+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:09.861+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:09.861+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:09.861+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:11.067 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:10.933+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:11.067 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:10.933+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:11.068 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:10.933+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:11.068 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:10.818+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:11.068 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:10.818+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:11.068 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:10.818+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:11.355 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:11.065+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:11.355 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:11.065+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:11.355 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:11.065+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:12.106 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:11.906+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:12.107 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:11.906+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:12.107 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:11.906+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:12.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:11.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:12.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:11.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:12.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:11.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:12.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:12.104+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:12.480 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:12.104+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:12.480 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:12.104+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:13.059+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:13.059+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:13.059+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:12.878+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:12.878+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:12.878+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:12.827+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:12.827+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:12.827+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:14.064 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:13.825+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:14.064 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:13.825+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:14.064 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:13.825+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:14.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:13.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:14.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:13.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:14.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:13.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:14.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:14.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:14.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:14.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:14.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:15.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:14.830+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:15.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:14.830+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:15.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:14.830+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:15.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:14.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:15.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:14.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:14.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:15.050+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:15.050+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:15.050+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:16.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:15.843+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:16.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:15.843+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:16.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:15.843+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:16.029 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:15.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:16.029 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:15.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:16.029 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:15.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:16.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:16.027+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:16.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:16.027+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:16.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:16.027+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:17.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:17.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:17.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:17.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:17.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:16.869+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:16.869+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:16.869+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:17.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:16.836+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:17.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:16.836+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:17.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:16.836+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:18.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:17.808+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:18.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:17.808+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:18.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:17.808+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:18.050+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:18.050+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:18.050+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:17.852+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:17.852+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:17.852+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:19.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:18.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:19.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:18.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:19.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:18.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:18.807+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:18.807+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:18.807+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:19.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:19.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:19.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:20.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:19.839+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:20.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:19.839+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:20.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:19.839+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:20.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:20.011+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:20.011+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:20.011+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:20.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:19.881+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:20.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:19.881+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:20.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:19.881+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:21.050 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:20.872+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:21.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:20.872+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:21.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:20.872+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:21.051 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:20.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:21.051 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:20.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:21.051 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:20.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:21.357 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:21.048+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:21.358 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:21.048+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:21.358 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:21.048+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:22.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:22.023+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:22.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:22.023+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:22.023+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:21.839+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:21.839+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:21.839+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:22.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:21.885+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:22.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:21.885+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:22.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:21.885+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:23.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:22.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:23.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:22.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:23.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:22.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:23.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:22.843+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:23.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:22.843+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:23.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:22.843+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:22.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:22.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:22.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:24.025 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:23.990+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:24.025 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:23.990+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:24.025 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:23.990+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:24.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:23.819+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:24.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:23.819+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:24.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:23.819+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:24.026 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:23.764+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:24.026 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:23.764+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:24.026 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:23.764+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:25.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:25.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:25.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:25.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:24.813+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:25.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:24.813+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:25.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:24.813+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:24.804+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:24.804+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:24.804+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:26.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:26.023+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:26.023+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:26.023+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:25.812+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:25.812+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:25.812+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:26.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:25.815+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:26.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:25.815+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:26.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:25.815+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:27.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:26.854+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:27.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:26.854+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:27.065 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:26.854+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:27.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:26.777+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:27.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:26.777+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:27.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:26.777+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:27.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:27.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:27.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:27.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:27.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:27.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:28.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:28.039+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:28.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:28.039+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:28.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:28.039+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:27.803+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:27.803+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:27.803+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:27.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:27.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:27.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:29.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:28.905+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:29.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:28.905+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:29.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:28.905+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:29.084 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:28.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:29.085 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:28.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:29.085 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:28.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:29.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:29.082+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:29.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:29.082+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:29.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:29.082+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:30.056+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:30.056+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:30.056+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:30.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:29.933+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:29.933+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:29.933+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:29.823+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:29.823+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:29.823+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:31.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:31.033+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:31.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:31.033+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:31.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:31.033+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:31.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:30.902+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:31.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:30.902+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:31.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:30.902+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:30.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:30.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:30.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:32.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:32.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:32.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:32.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:32.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:32.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:31.901+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:31.901+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:31.901+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:31.803+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:31.803+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:31.803+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:33.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:33.046+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:33.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:33.046+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:33.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:33.046+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:32.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:32.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:32.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:32.823+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:32.823+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:32.823+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:34.060 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:33.935+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:34.061 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:33.935+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:34.061 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:33.935+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:34.061 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:33.787+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:34.061 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:33.787+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:34.061 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:33.787+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:34.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:34.059+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:34.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:34.059+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:34.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:34.059+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:35.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:34.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:35.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:34.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:35.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:34.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:35.109 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:34.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:35.109 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:34.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:35.109 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:34.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:35.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:35.107+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:35.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:35.107+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:35.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:35.107+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:36.103 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:35.874+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:36.103 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:35.874+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:36.103 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:35.874+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:36.104 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:35.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:36.104 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:35.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:36.104 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:35.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:36.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:36.101+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:36.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:36.101+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:36.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:36.101+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:37.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:36.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:37.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:36.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:37.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:36.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:37.128 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:36.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:37.128 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:36.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:37.128 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:36.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:37.126+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:37.126+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:37.126+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:38.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:37.799+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:38.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:37.799+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:38.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:37.799+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:38.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:37.946+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:38.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:37.946+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:38.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:37.946+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:38.124+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:38.124+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:38.124+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:39.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:38.916+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:39.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:38.916+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:39.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:38.916+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:39.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:38.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:39.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:38.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:39.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:38.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:39.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:39.131+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:39.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:39.131+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:39.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:39.131+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:40.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:39.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:40.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:39.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:40.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:39.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:40.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:39.814+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:40.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:39.814+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:40.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:39.814+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:40.173+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:40.173+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:40.173+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:41.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:40.899+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:41.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:40.899+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:41.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:40.899+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:40.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:40.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:40.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:41.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:41.222+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:41.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:41.222+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:41.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:41.222+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:42.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:41.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:42.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:41.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:42.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:41.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:41.883+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:41.883+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:41.883+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:42.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:42.239+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:42.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:42.239+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:42.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:42.239+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:43.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:42.912+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:43.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:42.912+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:43.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:42.912+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:42.894+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:42.894+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:42.894+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:43.512 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:43.247+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:43.512 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:43.247+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:43.512 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:43.247+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:44.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:43.862+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:44.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:43.862+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:44.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:43.862+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:44.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:43.960+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:44.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:43.960+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:44.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:43.960+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:44.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:44.212+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:44.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:44.212+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:44.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:44.212+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:45.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:44.975+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:45.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:44.975+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:45.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:44.975+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:45.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:44.910+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:44.910+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:44.910+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:45.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:45.239+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:45.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:45.239+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:45.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:45.239+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:46.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:45.942+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:46.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:45.942+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:46.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:45.942+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:46.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:45.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:45.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:45.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:46.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:46.280+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:46.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:46.280+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:46.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:46.280+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:47.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:46.982+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:47.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:46.982+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:47.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:46.982+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:47.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:46.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:46.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:46.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:47.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:47.312+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:47.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:47.312+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:47.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:47.312+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:48.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:47.980+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:48.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:47.980+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:48.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:47.980+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:48.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:47.847+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:47.847+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:47.847+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:48.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:48.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:48.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:48.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:48.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:48.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:48.989+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:48.989+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:48.989+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:49.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:48.863+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:48.863+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:48.863+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:49.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:49.286+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:49.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:49.286+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:49.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:49.286+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:50.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:50.038+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:50.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:50.038+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:50.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:50.038+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:50.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:49.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:50.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:49.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:50.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:49.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:50.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:50.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:50.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:50.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:50.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:50.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:51.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:50.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:51.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:50.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:51.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:50.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:51.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:51.016+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:51.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:51.016+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:51.279 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:51.016+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:51.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:51.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:51.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:51.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:51.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:51.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:52.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:52.063+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:52.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:52.063+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:52.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:52.063+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:52.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:51.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:52.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:51.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:51.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:52.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:52.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:52.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:52.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:52.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:52.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:52.936+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:52.936+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:52.936+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:53.056+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:53.056+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:53.056+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:53.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:53.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:53.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:53.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:53.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:53.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:53.893+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:53.893+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:53.893+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:54.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:54.075+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:54.075+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:54.075+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:54.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:54.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:54.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:54.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:54.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:54.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:55.151 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:55.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:55.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:55.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:55.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:55.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:55.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:54.874+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:55.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:54.874+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:55.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:54.874+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:55.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:55.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:55.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:55.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:55.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:55.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:56.081 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:55.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:56.081 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:55.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:56.081 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:55.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:56.394 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:56.079+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:56.394 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:56.079+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:56.394 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:56.079+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:56.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:56.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:56.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:56.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:56.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:56.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:57.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:56.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:57.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:56.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:57.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:56.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:57.426 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:57.119+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:57.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:57.119+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:57.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:57.119+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:57.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:57.425+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:57.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:57.425+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:57.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:57.425+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:57.902+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:57.902+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:57.902+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:58.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:58.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:58.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:58.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:58.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:58.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:58.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:58.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:58.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:58.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:59.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:59.083+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:59.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:59.083+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:59.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:45:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:45:59.083+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:45:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:58.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:58.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:58.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:45:59.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:59.414+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:59.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:59.414+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:45:59.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:45:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:45:59.414+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:00.081 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:59.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:00.081 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:59.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:00.081 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:45:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:45:59.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:00.417 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:00.079+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:00.418 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:00.079+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:00.418 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:00.079+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:00.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:00.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:00.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:01.104 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:00.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:01.104 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:00.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:01.104 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:00.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:01.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:01.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:01.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:01.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:01.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:01.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:01.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:01.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:01.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:01.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:01.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:01.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:02.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:01.854+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:02.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:01.854+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:02.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:01.854+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:02.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:02.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:02.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:02.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:02.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:02.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:02.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:02.138+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:02.138+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:02.138+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:03.131 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:02.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:03.132 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:02.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:03.132 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:02.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:03.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:03.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:03.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:03.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:03.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:03.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:03.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:03.129+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:03.129+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:03.129+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:04.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:03.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:04.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:03.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:04.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:03.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:04.415 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:04.140+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:04.415 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:04.140+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:04.415 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:04.140+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:04.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:04.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:04.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:04.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:04.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:04.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:05.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:04.849+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:05.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:04.849+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:05.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:04.849+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:05.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:05.126+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:05.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:05.126+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:05.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:05.126+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:05.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:05.379+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:05.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:05.379+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:05.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:05.379+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:06.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:05.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:06.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:05.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:06.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:05.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:06.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:06.170+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:06.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:06.170+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:06.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:06.170+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:06.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:06.370+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:06.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:06.371+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:06.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:06.371+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:07.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:06.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:07.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:06.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:07.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:06.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:07.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:07.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:07.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:07.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:07.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:07.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:07.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:07.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:07.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:07.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:07.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:07.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:08.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:07.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:08.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:07.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:08.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:07.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:08.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:08.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:08.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:08.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:08.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:08.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:08.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:08.365+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:08.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:08.365+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:08.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:08.365+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:09.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:08.827+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:09.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:08.827+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:09.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:08.827+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:09.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:09.206+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:09.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:09.206+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:09.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:09.206+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:09.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:09.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:09.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:09.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:09.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:09.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:10.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:09.803+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:10.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:09.803+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:10.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:09.803+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:10.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:10.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:10.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:10.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:10.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:10.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:10.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:10.374+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:10.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:10.374+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:10.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:10.374+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:11.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:10.846+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:11.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:10.846+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:11.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:10.846+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:11.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:11.109+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:11.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:11.109+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:11.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:11.109+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:11.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:11.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:11.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:11.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:11.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:11.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:12.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:12.090+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:12.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:12.090+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:12.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:12.090+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:11.846+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:11.846+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:11.846+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:12.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:12.397+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:12.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:12.397+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:12.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:12.397+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:13.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:13.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:13.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:13.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:12.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:12.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:12.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:13.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:13.352+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:13.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:13.352+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:13.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:13.352+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:14.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:14.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:14.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:14.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:13.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:13.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:13.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:14.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:14.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:14.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:14.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:14.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:14.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:15.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:14.862+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:14.862+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:14.862+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:15.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:15.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:15.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:15.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:15.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:15.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:15.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:15.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:15.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:15.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:15.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:15.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:16.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:15.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:16.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:15.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:16.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:15.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:16.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:16.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:16.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:16.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:16.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:16.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:16.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:16.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:16.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:16.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:16.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:16.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:17.110 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:16.852+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:17.110 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:16.852+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:17.110 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:16.852+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:17.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:17.357+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:17.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:17.357+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:17.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:17.357+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:17.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:17.108+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:17.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:17.108+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:17.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:17.108+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:18.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:18.122+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:18.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:18.122+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:18.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:18.122+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:18.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:17.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:17.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:17.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:18.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:18.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:18.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:18.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:18.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:18.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:19.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:18.862+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:19.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:18.862+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:19.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:18.862+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:19.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:19.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:19.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:19.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:19.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:19.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:19.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:19.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:19.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:19.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:19.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:19.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:20.091 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:19.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:20.091 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:19.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:20.091 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:19.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:20.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:20.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:20.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:20.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:20.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:20.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:20.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:20.352+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:20.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:20.352+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:20.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:20.352+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:21.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:20.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:21.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:20.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:21.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:20.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:21.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:21.215+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:21.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:21.215+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:21.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:21.215+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:21.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:21.323+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:21.323+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:21.323+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:22.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:21.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:22.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:21.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:22.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:21.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:22.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:22.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:22.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:22.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:22.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:22.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:22.259+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:22.259+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:22.259+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:23.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:22.859+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:23.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:22.859+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:23.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:22.859+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:23.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:23.316+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:23.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:23.316+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:23.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:23.316+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:23.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:23.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:23.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:23.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:23.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:23.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:24.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:23.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:24.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:23.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:24.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:23.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:24.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:24.287+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:24.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:24.287+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:24.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:24.287+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:24.246+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:24.246+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:24.246+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:25.188 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:24.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:25.189 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:24.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:25.189 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:24.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:25.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:25.242+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:25.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:25.242+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:25.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:25.242+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:25.209+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:25.209+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:25.209+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:26.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:25.901+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:26.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:25.901+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:26.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:25.901+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:26.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:26.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:26.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:26.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:26.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:26.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:26.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:26.287+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:26.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:26.287+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:26.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:26.287+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:27.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:26.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:27.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:26.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:27.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:26.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:27.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:27.196+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:27.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:27.196+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:27.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:27.196+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:27.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:27.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:27.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:27.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:27.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:27.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:28.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:28.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:28.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:28.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:28.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:28.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:27.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:27.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:27.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:28.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:28.280+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:28.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:28.280+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:28.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:28.280+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:29.193 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:28.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:29.193 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:28.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:29.193 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:28.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:29.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:29.191+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:29.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:29.191+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:29.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:29.191+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:29.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:29.265+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:29.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:29.265+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:29.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:29.265+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:30.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:30.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:30.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:30.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:30.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:30.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:29.944+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:29.944+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:29.944+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:30.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:30.293+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:30.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:30.293+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:30.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:30.293+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:31.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:30.958+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:31.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:30.958+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:31.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:30.958+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:31.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:31.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:31.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:31.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:31.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:31.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:31.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:31.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:31.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:31.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:32.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:32.136+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:32.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:32.136+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:32.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:32.136+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:32.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:31.973+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:32.318 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:31.973+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:32.318 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:31.973+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:32.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:32.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:32.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:32.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:32.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:32.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:33.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:32.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:33.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:32.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:33.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:32.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:33.321 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:33.173+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:33.321 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:33.173+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:33.321 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:33.173+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:33.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:33.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:33.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:33.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:33.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:33.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:34.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:33.954+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:34.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:33.954+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:34.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:33.954+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:34.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:34.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:34.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:34.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:34.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:34.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:34.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:34.286+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:34.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:34.286+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:34.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:34.286+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:35.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:35.151+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:35.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:35.151+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:35.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:35.151+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:34.961+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:34.961+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:34.961+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:35.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:35.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:35.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:35.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:35.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:35.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:36.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:35.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:36.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:35.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:36.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:35.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:36.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:36.292+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:36.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:36.292+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:36.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:36.292+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:36.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:36.164+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:36.164+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:36.164+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:37.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:36.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:37.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:36.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:37.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:36.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:37.272+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:37.272+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:37.272+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:37.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:37.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:37.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:38.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:37.922+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:38.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:37.922+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:38.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:37.922+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:38.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:38.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:38.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:38.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:38.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:38.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:38.267+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:38.267+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:38.267+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:39.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:39.166+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:39.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:39.166+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:39.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:39.166+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:39.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:38.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:39.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:38.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:39.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:38.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:39.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:39.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:39.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:39.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:39.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:39.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:40.138+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:40.138+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:40.138+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:40.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:39.971+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:40.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:39.971+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:40.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:39.971+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:40.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:40.269+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:40.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:40.269+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:40.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:40.269+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:41.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:40.926+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:41.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:40.926+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:41.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:40.926+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:41.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:41.150+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:41.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:41.150+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:41.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:41.150+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:41.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:41.252+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:41.252+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:41.252+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:42.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:42.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:42.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:42.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:42.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:42.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:42.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:41.946+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:42.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:41.946+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:41.946+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:42.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:42.264+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:42.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:42.264+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:42.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:42.264+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:43.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:43.160+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:43.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:43.160+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:43.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:43.160+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:42.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:42.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:42.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:43.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:43.302+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:43.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:43.302+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:43.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:43.302+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:44.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:44.122+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:44.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:44.122+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:44.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:44.122+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:43.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:43.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:43.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:44.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:44.338+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:44.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:44.338+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:44.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:44.338+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:45.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:45.129+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:45.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:45.129+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:45.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:45.129+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:45.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:44.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:44.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:44.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:45.568 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:45.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:45.568 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:45.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:45.568 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:45.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:46.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:45.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:46.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:45.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:46.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:45.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:46.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:46.170+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:46.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:46.170+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:46.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:46.170+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:46.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:46.293+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:46.293+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:46.293+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:47.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:47.125+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:47.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:47.125+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:47.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:47.125+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:46.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:46.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:46.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:47.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:47.254+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:47.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:47.254+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:47.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:47.254+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:48.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:48.137+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:48.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:48.137+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:48.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:48.137+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:47.905+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:47.905+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:47.905+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:48.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:48.242+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:48.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:48.242+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:48.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:48.242+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:49.134+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:49.134+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:49.134+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:49.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:48.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:48.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:48.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:49.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:49.254+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:49.801 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:49.254+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:49.801 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:49.254+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:50.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:50.151+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:50.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:50.151+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:50.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:50.151+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:50.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:49.913+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:50.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:49.913+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:50.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:49.913+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:50.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:50.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:50.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:50.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:50.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:50.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:50.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:46:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:46:50.776+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:46:50.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:50.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:50.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:50.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:50.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:51.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:51.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:51.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:51.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:51.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:51.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:51.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:51.149+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:51.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:51.149+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:51.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:51.149+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:52.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:52.151+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:52.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:52.151+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:52.151+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:51.906+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:51.906+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:51.906+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:52.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:52.309+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:52.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:52.309+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:52.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:52.309+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:53.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:52.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:53.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:52.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:53.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:52.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:53.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:53.283+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:53.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:53.283+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:53.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:53.283+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:53.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:53.165+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:53.165+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:53.165+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:53.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:53.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:53.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:54.125+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:54.125+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:54.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:54.125+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:54.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:54.261+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:54.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:54.261+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:54.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:54.261+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:55.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:55.133+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:55.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:55.133+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:55.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:55.133+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:54.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:54.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:54.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:55.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:55.295+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:55.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:55.295+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:55.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:55.295+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:56.139 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:46:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:46:55.777+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 15 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:46:56.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:55.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:56.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:55.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:56.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:55.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:56.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:56.138+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:56.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:56.138+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:56.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:56.138+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:56.274+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:56.274+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:56.274+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:57.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:57.104+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:57.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:57.104+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:57.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:57.104+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:56.962+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:56.962+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:56.962+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:57.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:57.271+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:57.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:57.271+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:57.271+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:58.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:58.064+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:58.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:58.064+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:58.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:58.064+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:58.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:57.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:58.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:57.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:58.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:57.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:58.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:58.258+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:58.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:58.258+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:58.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:58.258+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:59.239 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:58.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:59.239 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:58.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:59.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:46:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:46:58.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:46:59.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:59.031+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:59.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:59.031+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:59.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:46:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:46:59.031+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:46:59.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:59.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:59.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:59.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:46:59.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:46:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:46:59.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:00.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:00.037+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:00.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:00.037+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:00.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:00.037+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:00.019+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:00.019+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:00.019+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:00.260+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:00.260+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:00.260+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:01.055 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:47:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:47:00.777+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 28 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:47:01.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:01.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:01.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:01.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:01.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:01.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:01.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:01.274+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:01.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:01.274+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:01.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:01.274+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:01.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:01.053+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:01.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:01.053+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:01.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:01.053+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:02.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:02.016+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:02.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:02.016+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:02.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:02.016+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:02.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:02.031+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:02.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:02.031+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:02.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:02.031+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:02.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:02.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:02.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:02.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:02.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:02.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:03.066+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:03.066+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:03.066+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:02.995+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:02.995+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:02.995+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:03.517 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:03.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:03.517 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:03.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:03.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:03.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:04.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:04.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:04.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:04.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:04.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:04.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:04.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:04.026+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:04.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:04.026+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:04.366 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:04.026+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:04.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:04.363+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:04.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:04.363+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:04.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:04.363+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:05.340 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:05.007+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:05.341 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:05.007+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:05.341 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:05.007+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:05.341 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:05.026+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:05.341 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:05.026+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:05.341 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:05.026+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:05.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:05.338+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:05.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:05.338+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:05.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:05.338+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:06.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:47:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:47:05.778+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 38 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:47:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:06.014+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:06.014+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:06.014+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:06.016+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:06.016+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:06.016+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:06.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:06.290+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:06.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:06.290+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:06.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:06.290+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:07.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:06.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:07.326 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:06.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:07.326 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:06.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:07.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:07.000+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:07.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:07.000+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:07.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:07.000+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:07.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:07.324+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:07.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:07.324+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:07.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:07.324+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:08.316 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:07.989+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:08.316 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:07.989+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:08.316 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:07.989+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:08.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:08.031+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:08.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:08.031+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:08.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:08.031+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:08.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:08.314+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:08.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:08.314+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:08.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:08.314+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:09.293 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:09.018+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:09.293 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:09.018+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:09.293 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:09.018+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:09.294 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:09.028+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:09.294 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:09.028+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:09.294 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:09.028+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:09.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:09.291+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:09.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:09.291+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:09.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:09.291+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:10.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:09.987+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:10.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:09.987+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:10.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:09.987+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:10.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:09.987+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:10.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:09.987+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:10.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:09.987+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:10.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:10.253+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:10.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:10.253+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:10.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:10.253+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:11.028 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:47:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:47:10.778+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 79 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:47:11.028 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:10.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:11.028 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:10.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:11.028 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:10.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:11.355 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:11.211+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:11.355 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:11.211+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:11.355 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:11.211+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:11.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:11.026+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:11.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:11.026+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:11.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:11.026+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:12.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:12.197+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:12.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:12.197+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:12.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:12.197+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:12.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:12.008+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:12.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:12.008+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:12.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:12.008+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:12.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:11.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:12.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:11.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:12.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:11.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:13.148+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:13.148+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:13.148+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:12.981+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:12.981+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:12.981+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:12.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:12.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:12.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:14.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:14.027+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:14.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:14.027+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:14.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:14.027+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:14.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:13.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:14.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:13.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:14.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:13.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:14.137+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:14.137+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:14.137+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:15.131+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:15.131+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:15.131+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:15.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:15.016+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:15.016+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:15.016+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:15.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:14.963+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:15.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:14.963+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:15.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:14.963+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:16.061 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:47:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:47:15.779+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 95 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:47:16.062 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:15.990+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:16.062 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:15.990+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:16.062 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:15.990+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:16.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:16.060+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:16.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:16.060+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:16.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:16.060+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:16.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:16.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:16.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:16.110+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:16.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:16.110+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:17.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:17.120+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:17.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:17.120+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:17.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:17.120+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:16.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:16.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:16.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:17.074+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:17.074+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:17.074+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:18.140+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:18.140+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:18.140+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:18.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:18.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:18.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:17.913+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:17.913+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:17.913+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:19.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:19.063+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:19.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:19.063+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:19.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:19.063+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:19.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:18.913+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:19.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:18.913+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:19.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:18.913+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:19.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:19.184+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:19.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:19.184+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:19.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:19.184+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:20.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:20.017+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:20.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:20.017+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:20.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:20.017+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:20.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:19.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:20.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:19.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:20.204 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:19.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:20.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:20.201+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:20.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:20.201+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:20.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:20.201+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:21.008 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:47:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:47:20.779+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 134 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:47:21.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:20.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:21.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:20.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:21.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:20.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:21.358 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:21.173+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:21.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:21.173+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:21.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:21.173+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:21.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:21.006+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:21.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:21.006+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:21.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:21.006+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:22.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:21.969+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:22.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:21.969+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:22.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:21.969+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:22.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:21.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:22.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:21.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:22.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:21.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:22.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:22.160+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:22.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:22.160+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:22.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:22.160+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:23.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:22.967+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:23.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:22.967+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:23.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:22.967+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:23.128 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:22.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:23.128 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:22.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:23.128 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:22.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:23.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:23.125+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:23.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:23.125+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:23.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:23.125+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:24.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:24.122+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:24.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:24.122+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:24.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:24.122+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:24.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:24.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:24.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:24.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:24.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:24.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:23.914+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:23.914+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:23.914+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:25.129+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:25.129+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:25.129+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:25.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:25.014+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:25.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:25.014+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:25.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:25.014+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:24.955+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:24.955+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:24.955+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:26.157 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:47:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:47:25.780+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 161 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:47:26.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:26.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:26.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:26.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:26.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:26.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:26.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:25.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:26.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:25.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:26.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:25.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:26.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:26.155+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:26.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:26.155+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:26.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:26.155+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:27.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:26.957+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:27.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:26.957+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:27.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:26.957+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:27.110+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:27.110+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:27.110+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:26.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:26.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:26.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:28.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:27.929+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:28.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:27.929+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:28.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:27.929+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:27.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:27.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:27.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:28.064+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:28.064+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:28.064+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:29.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:29.075+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:29.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:29.075+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:29.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:29.075+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:29.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:28.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:29.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:28.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:29.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:28.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:28.882+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:28.882+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:28.882+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:30.098+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:30.098+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:30.098+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:29.883+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:29.883+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:29.883+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:29.926+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:29.926+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:29.926+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:31.073 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:47:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:47:30.780+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 191 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:47:31.074 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:30.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:31.074 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:30.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:31.074 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:30.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:31.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:30.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:31.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:30.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:31.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:30.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:31.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:31.071+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:31.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:31.071+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:31.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:31.071+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:32.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:32.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:32.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:32.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:32.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:32.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:31.872+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:31.872+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:31.872+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:31.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:31.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:31.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:33.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:33.040+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:33.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:33.040+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:33.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:33.040+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:32.910+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:32.910+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:32.910+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:32.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:32.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:32.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:34.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:34.031+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:34.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:34.031+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:34.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:34.031+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:34.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:33.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:34.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:33.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:34.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:33.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:33.885+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:33.885+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:33.885+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:35.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:34.838+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:35.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:34.838+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:35.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:34.838+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:35.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:35.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:35.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:35.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:34.935+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:34.935+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:34.935+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:36.070 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:47:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:47:35.780+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 222 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:47:36.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:35.921+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:36.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:35.921+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:36.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:35.921+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:36.071 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:35.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:36.071 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:35.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:36.071 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:35.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:36.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:36.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:36.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:36.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:36.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:36.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:37.107 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:36.892+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:37.107 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:36.892+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:37.107 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:36.892+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:37.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:36.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:37.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:36.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:37.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:36.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:37.105+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:37.105+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:37.105+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:38.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:37.886+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:38.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:37.886+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:38.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:37.886+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:38.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:37.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:38.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:37.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:38.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:37.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:38.121+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:38.121+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:38.121+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:39.117+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:39.117+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:39.117+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:38.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:38.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:38.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:39.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:38.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:39.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:38.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:39.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:38.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:40.144 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:39.839+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:40.144 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:39.839+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:40.144 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:39.839+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:40.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:39.888+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:40.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:39.888+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:40.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:39.888+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:40.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:40.142+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:40.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:40.142+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:40.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:40.142+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:41.167 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:47:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:47:40.781+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 255 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:47:41.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:40.875+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:41.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:40.875+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:41.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:40.875+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:41.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:40.820+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:41.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:40.820+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:41.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:40.820+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:41.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:41.165+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:41.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:41.165+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:41.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:41.165+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:41.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:41.778+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:41.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:41.778+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:41.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:41.778+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:42.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:41.912+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:42.211 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:41.912+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:42.211 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:41.912+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:42.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:42.209+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:42.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:42.209+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:42.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:42.209+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:43.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:42.808+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:43.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:42.808+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:43.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:42.808+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:43.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:42.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:43.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:42.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:43.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:42.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:43.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:43.193+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:43.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:43.193+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:43.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:43.193+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:43.853 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:43.766+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:43.853 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:43.766+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:43.853 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:43.766+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:44.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:43.851+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:44.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:43.851+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:44.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:43.851+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:44.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:44.156+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:44.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:44.156+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:44.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:44.156+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:45.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:44.794+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:45.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:44.794+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:45.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:44.794+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:45.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:44.848+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:45.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:44.848+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:45.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:44.848+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:45.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:45.140+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:45.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:45.140+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:45.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:45.140+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:46.168 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:47:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:47:45.782+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 286 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:47:46.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:45.790+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:46.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:45.790+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:46.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:45.790+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:46.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:45.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:46.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:45.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:46.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:45.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:46.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:46.166+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:46.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:46.166+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:46.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:46.166+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:46.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:46.167+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:47:46.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:46.835+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:46.835+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:46.835+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:46.837+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:46.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:46.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:46.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:46.811+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:47.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:47.175+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:47.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:47.175+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:47.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:47.175+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:47.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:47.176+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:47:48.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:47.840+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:48.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:47.840+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:48.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:47.840+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:48.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:47.841+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:48.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:47.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:48.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:47.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:48.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:47.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:48.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:47.824+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:48.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:48.205+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:48.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:48.205+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:48.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:48.205+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:48.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:48.206+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:47:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:48.810+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:48.810+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:48.810+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:48.811+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:48.785+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:48.785+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:48.785+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:48.786+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:49.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:49.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:49.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:49.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:49.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:49.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:49.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:49.238+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:47:50.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:49.831+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:50.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:49.831+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:50.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:49.831+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:50.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:49.831+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:50.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:49.814+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:50.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:49.814+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:50.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:49.814+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:50.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:49.814+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:50.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:50.200+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:50.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:50.200+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:50.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:50.200+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:50.201+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:47:51.179 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:47:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:47:50.782+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 316 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:47:51.238 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:50.875+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:51.238 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:50.875+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:51.238 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:50.875+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:51.238 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:50.875+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:51.238 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:50.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:51.238 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:50.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:51.238 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:50.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:51.238 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:50.845+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:51.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:51.177+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:51.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:51.177+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:51.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:51.177+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:51.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:51.178+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:47:52.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:51.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:52.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:51.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:52.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:51.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:52.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:51.888+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:52.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:51.801+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:52.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:51.801+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:52.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:51.801+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:52.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:51.802+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:52.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:52.207+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:52.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:52.207+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:52.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:52.207+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:52.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:52.208+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:47:53.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:52.790+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:53.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:52.790+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:53.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:52.790+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:53.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:52.790+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:53.206 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:52.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:53.206 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:52.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:53.206 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:52.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:53.206 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:52.877+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:53.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:53.203+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:53.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:53.203+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:53.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:53.203+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:53.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:53.204+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:47:54.194 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:53.853+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:54.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:53.853+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:54.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:53.853+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:54.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:53.854+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:54.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:53.822+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:54.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:53.822+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:54.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:53.822+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:54.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:53.823+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:54.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:54.192+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:54.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:54.192+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:54.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:54.192+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:54.193+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:47:55.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:54.867+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:55.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:54.867+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:55.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:54.867+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:55.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:54.867+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:55.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:54.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:55.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:54.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:55.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:54.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:55.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:54.810+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:55.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:55.168+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:55.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:55.168+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:55.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:55.168+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:55.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:55.168+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:47:56.153 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:47:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:47:55.783+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 342 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:47:56.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:55.833+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:56.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:55.833+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:56.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:55.833+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:56.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:55.833+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:56.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:55.776+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:56.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:55.776+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:56.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:55.776+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:56.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:55.776+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:56.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:56.150+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:56.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:56.150+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:56.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:56.150+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:56.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:56.151+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:47:57.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:56.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:57.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:56.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:57.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:56.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:57.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:56.847+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:57.132 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:56.730+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:57.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:56.730+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:57.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:56.730+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:57.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:56.730+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:57.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:57.130+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:57.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:57.130+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:57.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:57.130+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:57.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:57.131+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:47:58.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:57.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:58.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:57.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:58.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:57.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:58.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:57.897+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:58.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:57.771+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:58.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:57.771+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:58.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:57.771+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:58.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:57.771+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:58.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:58.154+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:58.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:58.154+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:58.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:58.154+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:58.155+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:47:59.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:58.855+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:59.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:58.855+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:59.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:58.855+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:47:59.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:58.855+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:59.188 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:58.814+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:59.188 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:58.814+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:59.188 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:58.814+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:47:59.188 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:58.815+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:47:59.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:59.185+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:59.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:59.185+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:59.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:59.185+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:47:59.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:47:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:47:59.186+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:00.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:59.808+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:00.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:59.808+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:00.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:59.808+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:00.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:47:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:47:59.809+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:00.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:59.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:00.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:59.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:00.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:59.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:00.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:47:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:47:59.832+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:00.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:00.184+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:00.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:00.184+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:00.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:00.184+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:00.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:00.185+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:01.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:48:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:48:00.783+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 373 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:48:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:00.799+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:00.799+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:00.799+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:00.799+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:01.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:00.818+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:01.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:00.818+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:01.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:00.818+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:01.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:00.819+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:01.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:01.227+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:01.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:01.227+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:01.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:01.227+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:01.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:01.228+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:01.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:01.771+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:01.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:01.771+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:01.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:01.771+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:01.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:01.771+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:01.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:01.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:01.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:01.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:01.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:01.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:01.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:01.753+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:02.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:02.185+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:02.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:02.185+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:02.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:02.185+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:02.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:02.185+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:03.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:02.741+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:03.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:02.741+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:03.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:02.741+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:03.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:02.741+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:03.209 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:02.800+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:03.209 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:02.800+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:03.209 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:02.800+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:03.209 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:02.800+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:03.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:03.207+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:03.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:03.207+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:03.207+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:03.207+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:03.774 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:03.710+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:03.774 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:03.710+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:03.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:03.710+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:03.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:03.711+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:03.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:03.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:03.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:03.773+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:04.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:04.230+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:04.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:04.230+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:04.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:04.230+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:04.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:04.230+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:04.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:04.704+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:04.704+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:04.704+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:04.705+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:04.758+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:04.758+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:04.758+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:04.758+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:05.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:05.211+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:05.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:05.211+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:05.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:05.211+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:05.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:05.212+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:06.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:48:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:48:05.784+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 400 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:48:06.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:05.730+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:05.730+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:05.730+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:05.730+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:05.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:05.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:05.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:05.748+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:06.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:06.238+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:06.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:06.238+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:06.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:06.238+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:06.239+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:07.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:06.753+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:07.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:06.753+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:07.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:06.753+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:07.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:06.754+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:07.092 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:06.764+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:07.092 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:06.764+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:07.092 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:06.764+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:07.093 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:06.764+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:07.440 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:07.287+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:07.441 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:07.287+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:07.441 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:07.287+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:07.441 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:07.288+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:07.739+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:07.739+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:07.739+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:07.739+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:08.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:07.719+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:08.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:07.719+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:08.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:07.719+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:08.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:07.720+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:08.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:08.240+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:08.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:08.240+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:08.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:08.240+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:08.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:08.241+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:08.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:08.708+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:08.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:08.708+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:08.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:08.708+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:08.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:08.709+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:08.732+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:08.732+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:08.732+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:08.733+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:09.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:09.283+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:09.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:09.283+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:09.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:09.283+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:09.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:09.284+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:10.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:09.726+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:10.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:09.726+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:10.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:09.726+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:09.727+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:09.778+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:09.778+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:09.778+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:09.779+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:10.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:10.282+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:10.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:10.282+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:10.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:10.282+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:10.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:10.283+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:10.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:48:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:48:10.785+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 430 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:48:10.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:10.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:10.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:10.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:10.694+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:10.749+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:10.749+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:10.749+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:10.750+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:11.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:11.327+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:11.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:11.327+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:11.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:11.327+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:11.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:11.328+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:11.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:11.666+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:11.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:11.666+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:11.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:11.666+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:11.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:11.667+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:11.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:11.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:11.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:11.780+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:12.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:12.310+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:12.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:12.310+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:12.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:12.310+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:12.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:12.310+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:12.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:12.647+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:12.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:12.647+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:12.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:12.647+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:12.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:12.647+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:12.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:12.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:12.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:12.753+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:13.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:13.271+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:13.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:13.271+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:13.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:13.271+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:13.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:13.272+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:13.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:13.665+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:13.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:13.665+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:13.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:13.665+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:13.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:13.666+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:13.732+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:13.732+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:13.732+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:13.733+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:14.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:14.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:14.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:14.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:14.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:14.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:14.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:14.312+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:14.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:14.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:14.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:14.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:14.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:14.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:14.665+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:14.708+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:14.708+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:14.708+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:14.708+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:15.663 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:15.305+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:15.663 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:15.305+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:15.663 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:15.305+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:15.663 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:15.306+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:15.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:48:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:48:15.785+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 456 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:48:15.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:15.707+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:15.707+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:15.707+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:15.708+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:15.661+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:15.661+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:15.661+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:15.662+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:16.650 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:16.290+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:16.650 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:16.290+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:16.650 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:16.290+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:16.650 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:16.291+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:16.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:16.648+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:16.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:16.648+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:16.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:16.648+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:16.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:16.648+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:16.669+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:16.669+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:16.669+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:16.670+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:17.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:17.304+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:17.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:17.304+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:17.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:17.304+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:17.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:17.305+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:17.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:17.687+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:17.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:17.687+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:17.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:17.687+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:17.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:17.688+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:17.688+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:17.688+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:17.688+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:17.689+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:18.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:18.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:18.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:18.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:18.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:18.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:18.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:18.344+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:18.681+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:18.681+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:18.681+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:18.682+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:18.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:18.658+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:18.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:18.658+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:18.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:18.658+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:18.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:18.659+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:19.683 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:19.323+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:19.683 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:19.323+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:19.684 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:19.323+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:19.684 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:19.323+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:19.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:19.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:19.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:19.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:19.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:19.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:19.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:19.692+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:19.680+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:19.680+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:19.680+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:19.681+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:20.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:20.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:20.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:20.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:20.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:20.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:20.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:20.341+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:20.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:48:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:48:20.785+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 485 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:48:20.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:20.693+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:20.693+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:20.693+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:20.694+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:20.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:20.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:20.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:20.712+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:21.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:21.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:21.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:21.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:21.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:21.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:21.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:21.356+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:21.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:21.649+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:21.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:21.649+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:21.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:21.649+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:21.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:21.650+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:21.678+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:21.678+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:21.678+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:21.679+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:22.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:22.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:22.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:22.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:22.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:22.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:22.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:22.352+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:22.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:22.719+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:22.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:22.719+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:22.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:22.719+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:22.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:22.719+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:22.642+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:22.642+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:22.642+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:22.642+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:23.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:23.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:23.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:23.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:23.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:23.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:23.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:23.342+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:23.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:23.735+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:23.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:23.735+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:23.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:23.735+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:23.736+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:23.642+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:23.642+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:23.642+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:23.643+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:24.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:24.296+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:24.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:24.296+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:24.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:24.296+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:24.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:24.297+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:24.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:24.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:24.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:24.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:24.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:24.688+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:24.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:24.610+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:24.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:24.610+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:24.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:24.610+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:24.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:24.610+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:25.624 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:25.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:25.624 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:25.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:25.624 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:25.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:25.624 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:25.343+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:25.963 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:48:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:48:25.786+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 512 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:48:25.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:25.621+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:25.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:25.621+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:25.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:25.621+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:25.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:25.622+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:25.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:25.696+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:25.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:25.696+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:25.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:25.696+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:25.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:25.697+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:26.594 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:26.318+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:26.594 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:26.318+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:26.594 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:26.318+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:26.594 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:26.319+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:26.595 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:26.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:26.595 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:26.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:26.595 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:26.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:26.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:26.593+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:26.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:26.707+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:26.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:26.707+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:26.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:26.707+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:26.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:26.708+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:27.565 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:27.272+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:27.565 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:27.272+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:27.565 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:27.272+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:27.565 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:27.273+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:27.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:27.563+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:27.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:27.563+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:27.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:27.563+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:27.563+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:27.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:27.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:27.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:27.693+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:28.573 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:28.293+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:28.573 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:28.293+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:28.573 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:28.293+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:28.573 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:28.294+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:28.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:28.738+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:28.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:28.738+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:28.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:28.738+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:28.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:28.739+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:28.945 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:28.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:28.945 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:28.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:28.945 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:28.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:28.945 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:28.571+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:29.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:29.263+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:29.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:29.263+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:29.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:29.263+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:29.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:29.263+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:29.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:29.565+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:29.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:29.565+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:29.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:29.565+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:29.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:29.565+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:29.739+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:29.740+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:29.740+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:29.740+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:30.545 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:30.270+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:30.545 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:30.270+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:30.545 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:30.270+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:30.545 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:30.271+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:30.800 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:48:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:48:30.787+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 546 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:48:30.800 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:30.746+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:30.800 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:30.746+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:30.800 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:30.746+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:30.800 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:30.747+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:30.800 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:30.543+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:30.801 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:30.543+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:30.801 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:30.543+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:30.801 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:30.543+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:31.547 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:31.247+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:31.547 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:31.247+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:31.547 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:31.247+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:31.547 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:31.247+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:31.718+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:31.718+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:31.718+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:31.719+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:31.545+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:31.545+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:31.545+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:31.546+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:32.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:32.250+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:32.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:32.250+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:32.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:32.250+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:32.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:32.251+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:32.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:32.681+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:32.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:32.681+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:32.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:32.681+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:32.681+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:32.581+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:32.581+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:32.581+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:32.582+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:33.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:33.216+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:33.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:33.216+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:33.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:33.216+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:33.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:33.216+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:33.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:33.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:33.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:33.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:33.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:33.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:33.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:33.650+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:33.569+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:33.569+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:33.569+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:33.570+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:34.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:34.189+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:34.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:34.189+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:34.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:34.189+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:34.190+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:34.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:34.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:34.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:34.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:34.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:34.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:34.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:34.651+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:34.601+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:34.601+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:34.601+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:34.601+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:35.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:35.201+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:35.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:35.201+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:35.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:35.201+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:35.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:35.201+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:35.860 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:48:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:48:35.787+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 574 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:48:35.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:35.607+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:35.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:35.607+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:35.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:35.607+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:35.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:35.607+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:35.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:35.640+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:35.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:35.640+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:35.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:35.640+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:35.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:35.640+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:36.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:36.234+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:36.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:36.234+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:36.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:36.234+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:36.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:36.234+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:36.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:36.641+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:36.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:36.641+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:36.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:36.641+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:36.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:36.642+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:36.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:36.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:36.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:36.621+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:37.613 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:37.251+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:37.620 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:37.251+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:37.620 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:37.251+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:37.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:37.252+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:37.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:37.627+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:37.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:37.627+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:37.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:37.627+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:37.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:37.628+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:37.611+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:37.611+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:37.611+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:37.612+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:38.204+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:38.204+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:38.204+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:38.204+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:38.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:38.661+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:38.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:38.661+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:38.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:38.661+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:38.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:38.662+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:38.623+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:38.623+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:38.623+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:38.624+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:39.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:39.178+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:39.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:39.178+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:39.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:39.178+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:39.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:39.179+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:39.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:39.682+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:39.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:39.682+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:39.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:39.682+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:39.682+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:39.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:39.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:39.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:39.587+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:40.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:40.135+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:40.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:40.135+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:40.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:40.135+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:40.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:40.135+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:40.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:48:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:48:40.788+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 603 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:48:40.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:40.673+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:40.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:40.673+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:40.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:40.673+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:40.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:40.674+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:40.564+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:40.564+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:40.564+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:40.564+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:41.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:41.108+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:41.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:41.108+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:41.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:41.108+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:41.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:41.108+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:41.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:41.523+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:41.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:41.523+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:41.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:41.523+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:41.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:41.524+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:41.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:41.723+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:41.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:41.723+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:41.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:41.723+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:41.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:41.724+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:42.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:42.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:42.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:42.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:42.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:42.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:42.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:42.110+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:42.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:42.489+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:42.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:42.489+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:42.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:42.489+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:42.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:42.490+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:42.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:42.681+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:42.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:42.681+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:42.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:42.681+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:42.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:42.682+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:43.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:43.095+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:43.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:43.095+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:43.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:43.095+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:43.096+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:43.662+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:43.662+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:43.662+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:43.662+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:43.468+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:43.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:43.468+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:43.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:43.468+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:43.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:43.469+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:44.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:44.050+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:44.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:44.050+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:44.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:44.050+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:44.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:44.051+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:44.652 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:44.432+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:44.652 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:44.432+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:44.652 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:44.432+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:44.652 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:44.433+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:44.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:44.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:44.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:44.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:44.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:44.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:44.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:44.651+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:45.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:45.095+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:45.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:45.095+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:45.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:45.095+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:45.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:45.096+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:45.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:45.415+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:45.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:45.415+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:45.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:45.415+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:45.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:45.416+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:45.964 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:48:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:48:45.788+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 630 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:48:45.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:45.692+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:45.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:45.692+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:45.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:45.692+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:45.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:45.693+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:46.080+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:46.080+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:46.080+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:46.080+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:46.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:46.672+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:46.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:46.672+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:46.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:46.672+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:46.673+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:46.424+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:46.424+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:46.424+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:46.424+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:47.395 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:47.127+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:47.395 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:47.127+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:47.396 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:47.127+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:47.396 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:47.128+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:47.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:47.393+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:47.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:47.393+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:47.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:47.393+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:47.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:47.394+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:47.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:47.672+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:47.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:47.672+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:47.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:47.672+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:47.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:47.672+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:48.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:48.173+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:48.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:48.173+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:48.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:48.173+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:48.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:48.174+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:48.351+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:48.351+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:48.351+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:48.352+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:48.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:48.713+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:48.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:48.713+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:48.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:48.713+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:48.713+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:49.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:49.311+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:49.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:49.311+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:49.311+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:49.312+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:49.180+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:49.180+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:49.180+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:49.181+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:50.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:49.750+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:50.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:49.750+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:50.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:49.750+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:50.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:49.750+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:50.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:50.189+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:50.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:50.189+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:50.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:50.189+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:50.189+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:50.342+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:50.342+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:50.342+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:50.343+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:51.209 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:48:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:48:50.789+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 660 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:48:51.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:50.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:51.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:50.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:51.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:50.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:51.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:50.714+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:51.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:51.207+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:51.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:51.207+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:51.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:51.207+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:51.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:51.208+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:51.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:51.362+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:51.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:51.362+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:51.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:51.362+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:51.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:51.363+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:51.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:51.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:51.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:51.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:51.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:51.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:51.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:51.672+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:52.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:52.175+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:52.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:52.175+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:52.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:52.175+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:52.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:52.175+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:52.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:52.355+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:52.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:52.355+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:52.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:52.355+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:52.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:52.356+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:53.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:52.715+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:53.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:52.715+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:53.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:52.715+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:53.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:52.715+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:53.405 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:53.130+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:53.405 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:53.130+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:53.405 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:53.130+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:53.405 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:53.131+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:53.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:53.403+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:53.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:53.403+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:53.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:53.403+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:53.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:53.403+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:53.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:53.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:53.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:53.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:53.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:53.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:53.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:53.712+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:54.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:54.383+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:54.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:54.383+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:54.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:54.383+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:54.384+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:54.175+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:54.175+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:54.175+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:54.176+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:54.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:54.682+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:54.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:54.682+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:54.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:54.682+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:54.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:54.683+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:55.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:55.129+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:55.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:55.129+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:55.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:55.129+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:55.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:55.130+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:55.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:55.380+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:55.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:55.380+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:55.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:55.380+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:55.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:55.380+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:55.964 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:48:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:48:55.789+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 691 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:48:55.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:55.675+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:55.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:55.675+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:55.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:55.675+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:55.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:55.676+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:56.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:56.099+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:56.099+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:56.099+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:56.099+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:56.672 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:56.368+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:56.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:56.368+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:56.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:56.368+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:56.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:56.369+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:56.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:56.670+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:56.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:56.670+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:56.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:56.670+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:56.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:56.671+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:57.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:57.060+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:57.355 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:57.060+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:57.355 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:57.060+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:57.355 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:57.060+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:57.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:57.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:57.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:57.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:57.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:57.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:57.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:57.353+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:57.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:57.672+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:57.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:57.672+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:57.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:57.672+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:57.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:57.673+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:58.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:58.019+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:58.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:58.019+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:58.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:58.019+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:58.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:58.020+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:58.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:58.361+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:58.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:58.361+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:58.690 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:58.361+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:58.690 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:58.362+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:58.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:58.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:58.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:58.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:58.688+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:59.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:59.049+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:59.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:59.049+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:59.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:59.049+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:48:59.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:48:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:48:59.050+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:48:59.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:59.386+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:59.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:59.386+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:59.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:59.386+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:48:59.679 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:48:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:48:59.387+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:48:59.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:59.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:59.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:59.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:59.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:59.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:48:59.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:48:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:48:59.677+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:00.387 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:00.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:00.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:00.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:00.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:00.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:00.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:00.016+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:00.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:00.386+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:00.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:00.386+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:00.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:00.386+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:00.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:00.386+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:00.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:49:00.790+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 735 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:49:00.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:00.679+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:00.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:00.679+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:00.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:00.679+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:00.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:00.679+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:01.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:00.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:01.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:00.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:01.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:00.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:01.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:00.998+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:01.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:01.425+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:01.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:01.425+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:01.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:01.425+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:01.687 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:01.426+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:01.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:01.684+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:01.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:01.684+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:01.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:01.684+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:01.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:01.685+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:02.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:01.965+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:02.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:01.965+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:02.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:01.965+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:02.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:01.966+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:02.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:02.442+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:02.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:02.442+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:02.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:02.442+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:02.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:02.442+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:02.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:02.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:02.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:02.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:02.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:02.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:02.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:02.639+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:03.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:02.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:03.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:02.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:02.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:02.993+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:03.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:03.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:03.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:03.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:03.478+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:03.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:03.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:03.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:03.651+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:04.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:04.042+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:04.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:04.042+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:04.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:04.042+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:04.042+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:04.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:04.653+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:04.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:04.653+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:04.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:04.653+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:04.654+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:04.507+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:04.507+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:04.507+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:04.508+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:05.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:05.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:05.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:05.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:05.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:05.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:05.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:05.007+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:05.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:05.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:05.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:05.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:05.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:05.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:05.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:05.640+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:05.793 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:05.506+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:05.793 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:05.506+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:05.793 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:05.506+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:05.793 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:05.506+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:06.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:49:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:49:05.789+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 762 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:49:06.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:06.042+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:06.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:06.042+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:06.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:06.042+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:06.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:06.042+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:06.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:06.550+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:06.550+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:06.550+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:06.551+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:06.634+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:06.634+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:06.634+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:06.635+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:07.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:07.012+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:07.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:07.012+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:07.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:07.012+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:07.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:07.013+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:07.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:07.520+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:07.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:07.520+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:07.520+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:07.521+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:07.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:07.645+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:07.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:07.645+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:07.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:07.645+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:07.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:07.645+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:08.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:08.007+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:08.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:08.007+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:08.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:08.007+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:08.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:08.008+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:08.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:08.641+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:08.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:08.641+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:08.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:08.641+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:08.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:08.642+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:08.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:08.568+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:08.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:08.568+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:08.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:08.568+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:08.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:08.569+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:09.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:09.058+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:09.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:09.058+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:09.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:09.058+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:09.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:09.059+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:09.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:09.655+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:09.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:09.655+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:09.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:09.655+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:09.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:09.656+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:09.607+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:09.607+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:09.607+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:09.607+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:10.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:10.078+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:10.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:10.078+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:10.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:10.078+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:10.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:10.079+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:10.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:49:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:49:10.791+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 792 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:49:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:10.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:10.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:10.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:10.617+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:10.623+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:10.623+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:10.623+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:10.624+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:11.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:11.128+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:11.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:11.128+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:11.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:11.128+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:11.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:11.129+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:11.953 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:11.624+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:11.953 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:11.624+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:11.953 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:11.624+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:11.953 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:11.625+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:11.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:11.585+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:11.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:11.585+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:11.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:11.585+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:11.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:11.586+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:12.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:12.156+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:12.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:12.156+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:12.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:12.156+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:12.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:12.157+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:12.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:12.616+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:12.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:12.616+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:12.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:12.616+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:12.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:12.617+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:12.629+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:12.629+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:12.629+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:12.630+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:13.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:13.176+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:13.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:13.176+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:13.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:13.176+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:13.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:13.176+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:13.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:13.584+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:13.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:13.584+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:13.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:13.584+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:13.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:13.585+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:13.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:13.652+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:13.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:13.652+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:13.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:13.652+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:13.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:13.653+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:14.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:14.169+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:14.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:14.169+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:14.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:14.169+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:14.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:14.169+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:14.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:14.540+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:14.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:14.540+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:14.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:14.540+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:14.541+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:14.671+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:14.671+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:14.671+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:14.672+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:15.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:15.157+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:15.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:15.157+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:15.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:15.157+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:15.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:15.158+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:15.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:49:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:49:15.791+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 818 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:49:15.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:15.571+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:15.876 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:15.571+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:15.876 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:15.571+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:15.876 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:15.572+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:15.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:15.686+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:15.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:15.686+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:15.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:15.686+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:15.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:15.687+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:16.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:16.174+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:16.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:16.174+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:16.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:16.174+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:16.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:16.175+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:16.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:16.565+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:16.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:16.565+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:16.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:16.565+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:16.566+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:16.653+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:16.653+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:16.653+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:16.654+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:17.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:17.205+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:17.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:17.205+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:17.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:17.205+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:17.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:17.206+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:17.583+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:17.583+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:17.583+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:17.584+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:17.648+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:17.648+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:17.648+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:17.648+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:18.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:18.185+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:18.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:18.185+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:18.185+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:18.185+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:18.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:18.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:18.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:18.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:18.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:18.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:18.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:18.554+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:18.641+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:18.641+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:18.641+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:18.641+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:19.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:19.192+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:19.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:19.192+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:19.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:19.192+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:19.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:19.193+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:19.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:19.594+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:19.594+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:19.594+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:19.595+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:19.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:19.597+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:19.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:19.597+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:19.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:19.597+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:19.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:19.598+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:20.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:20.159+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:20.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:20.159+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:20.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:20.159+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:20.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:20.160+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:20.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:49:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:49:20.792+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 847 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:49:20.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:20.601+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:20.601+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:20.601+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:20.602+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:20.567+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:20.567+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:20.567+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:20.568+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:21.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:21.154+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:21.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:21.154+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:21.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:21.154+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:21.155+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:21.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:21.635+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:21.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:21.635+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:21.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:21.635+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:21.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:21.635+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:21.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:21.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:21.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:21.610+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:22.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:22.122+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:22.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:22.122+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:22.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:22.122+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:22.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:22.122+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:22.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:22.657+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:22.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:22.657+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:22.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:22.657+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:22.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:22.657+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:22.606+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:22.606+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:22.606+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:22.607+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:23.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:23.078+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:23.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:23.078+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:23.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:23.078+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:23.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:23.079+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:23.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:23.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:23.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:23.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:23.621+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:23.633+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:23.633+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:23.633+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:23.634+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:24.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:24.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:24.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:24.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:24.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:24.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:24.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:24.053+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:24.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:24.589+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:24.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:24.589+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:24.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:24.589+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:24.589+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:24.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:24.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:24.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:24.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:24.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:24.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:24.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:24.588+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:25.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:25.035+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:25.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:25.035+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:25.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:25.035+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:25.035+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:25.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:49:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:49:25.792+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 874 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:49:25.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:25.608+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:25.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:25.608+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:25.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:25.608+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:25.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:25.609+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:25.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:25.594+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:25.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:25.594+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:25.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:25.594+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:25.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:25.595+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:26.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:26.031+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:26.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:26.031+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:26.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:26.031+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:26.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:26.031+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:26.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:26.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:26.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:26.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:26.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:26.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:26.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:26.604+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:26.611+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:26.611+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:26.611+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:26.611+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:27.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:26.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:27.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:26.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:27.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:26.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:27.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:26.997+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:27.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:27.572+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:27.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:27.572+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:27.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:27.572+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:27.572+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:27.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:27.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:27.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:27.589+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:28.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:27.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:28.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:27.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:28.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:27.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:28.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:27.972+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:28.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:28.590+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:28.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:28.590+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:28.590+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:28.590+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:28.635+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:28.635+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:28.635+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:28.636+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:29.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:28.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:29.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:28.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:29.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:28.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:29.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:28.997+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:29.951 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:29.607+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:29.951 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:29.607+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:29.951 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:29.607+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:29.951 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:29.607+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:29.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:29.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:29.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:29.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:29.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:29.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:29.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:29.569+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:29.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:29.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:29.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:29.949+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:30.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:30.530+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:30.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:30.530+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:30.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:30.530+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:30.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:30.531+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:30.935 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:49:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:49:30.793+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 904 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:49:30.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:30.651+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:30.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:30.651+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:30.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:30.651+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:30.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:30.652+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:31.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:30.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:31.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:30.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:31.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:30.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:31.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:30.934+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:31.697 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:31.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:31.697 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:31.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:31.697 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:31.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:31.697 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:31.489+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:31.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:31.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:31.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:31.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:31.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:31.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:31.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:31.916+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:31.695+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:31.695+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:31.695+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:31.696+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:32.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:32.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:32.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:32.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:32.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:32.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:32.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:32.482+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:32.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:32.700+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:32.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:32.700+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:32.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:32.700+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:32.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:32.701+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:33.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:32.866+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:33.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:32.866+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:33.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:32.866+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:33.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:32.867+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:33.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:33.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:33.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:33.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:33.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:33.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:33.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:33.461+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:34.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:33.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:34.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:33.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:34.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:33.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:33.728+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:34.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:33.859+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:34.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:33.859+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:34.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:33.859+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:34.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:33.859+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:34.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:34.427+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:34.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:34.427+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:34.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:34.427+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:34.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:34.427+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:35.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:34.886+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:35.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:34.886+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:35.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:34.886+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:34.887+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:34.732+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:34.732+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:34.732+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:34.733+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:35.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:35.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:35.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:35.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:35.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:35.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:35.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:35.425+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:36.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:49:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:49:35.793+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 930 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:49:36.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:35.753+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:35.753+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:35.753+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:35.754+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:35.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:35.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:35.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:35.931+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:36.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:36.438+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:36.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:36.438+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:36.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:36.438+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:36.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:36.439+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:36.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:36.739+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:36.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:36.739+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:36.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:36.739+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:36.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:36.739+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:36.963+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:36.963+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:36.963+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:37.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:36.964+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:37.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:37.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:37.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:37.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:37.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:37.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:37.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:37.482+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:38.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:37.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:38.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:37.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:37.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:37.959+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:37.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:37.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:37.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:37.751+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:38.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:38.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:38.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:38.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:38.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:38.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:38.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:38.490+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:38.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:38.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:38.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:38.962+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:38.797+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:38.797+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:38.797+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:38.798+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:39.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:39.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:39.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:39.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:39.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:39.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:39.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:39.469+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:40.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:39.998+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:40.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:39.998+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:39.998+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:39.999+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:39.773+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:39.773+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:39.773+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:39.774+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:40.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:40.511+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:40.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:40.511+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:40.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:40.511+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:40.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:40.511+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:49:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:49:40.794+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 959 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:49:41.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:40.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:40.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:40.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:40.791+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:40.987+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:40.987+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:40.987+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:40.987+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:41.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:41.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:41.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:41.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:41.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:41.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:41.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:41.490+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:42.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:42.019+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:42.019+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:42.019+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:42.019+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:41.836+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:41.836+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:42.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:41.836+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:42.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:41.837+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:42.833 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:42.533+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:42.833 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:42.533+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:42.833 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:42.533+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:42.834 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:42.534+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:43.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:42.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:43.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:42.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:43.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:42.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:42.832+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:43.003+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:43.003+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:43.003+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:43.004+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:43.818 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:43.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:43.818 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:43.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:43.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:43.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:43.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:43.506+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:44.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:44.034+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:44.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:44.034+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:44.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:44.034+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:44.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:44.035+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:43.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:43.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:43.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:43.817+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:44.856 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:44.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:44.856 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:44.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:44.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:44.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:44.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:44.541+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:45.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:45.039+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:45.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:45.039+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:45.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:45.039+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:45.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:45.040+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:44.853+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:44.853+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:44.853+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:44.854+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:45.806 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:49:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:49:45.795+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 986 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:49:45.807 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:45.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:45.807 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:45.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:45.807 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:45.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:45.807 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:45.550+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:46.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:45.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:46.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:45.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:46.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:45.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:46.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:45.881+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:46.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:46.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:46.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:46.018+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:46.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:46.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:46.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:46.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:46.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:46.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:46.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:46.578+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:47.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:47.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:47.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:47.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:47.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:47.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:47.063+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:46.864+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:46.864+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:46.864+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:46.865+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:47.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:47.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:47.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:47.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:47.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:47.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:47.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:47.604+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:48.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:48.040+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:48.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:48.040+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:48.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:48.040+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:48.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:48.041+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:47.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:47.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:47.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:47.882+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:48.859 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:48.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:48.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:48.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:48.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:48.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:48.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:48.563+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:49.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:48.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:49.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:48.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:49.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:48.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:48.859+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:49.076+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:49.076+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:49.076+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:49.077+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:49.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:49.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:49.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:49.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:49.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:49.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:49.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:49.550+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:50.115+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:50.115+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:50.115+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:50.115+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:50.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:49.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:50.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:49.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:50.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:49.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:50.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:49.867+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:50.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:49:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:49:50.794+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 1031 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:49:50.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:50.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:50.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:50.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:50.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:50.836+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:50.591+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:50.591+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:50.591+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:50.592+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:51.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:51.101+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:51.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:51.101+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:51.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:51.101+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:51.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:51.101+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:51.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:51.575+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:51.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:51.575+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:51.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:51.575+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:51.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:51.576+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:52.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:51.798+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:52.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:51.798+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:52.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:51.798+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:52.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:51.799+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:52.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:52.151+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:52.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:52.151+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:52.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:52.151+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:52.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:52.152+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:52.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:52.814+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:52.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:52.814+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:52.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:52.814+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:52.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:52.815+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:52.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:52.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:52.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:52.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:52.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:52.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:52.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:52.574+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:53.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:53.174+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:53.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:53.174+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:53.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:53.174+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:53.174+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:53.795 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:53.567+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:53.795 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:53.567+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:53.795 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:53.567+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:53.795 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:53.567+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:54.064 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:53.793+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:54.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:53.793+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:54.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:53.793+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:54.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:53.793+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:54.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:54.140+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:54.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:54.140+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:54.140+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:54.141+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:54.836 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:54.538+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:54.836 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:54.538+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:54.836 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:54.538+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:54.836 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:54.539+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:55.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:54.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:55.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:54.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:55.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:54.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:55.115 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:54.834+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:55.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:55.112+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:55.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:55.112+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:55.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:55.112+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:55.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:55.113+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:55.810 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:49:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:49:55.796+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 1074 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:49:55.810 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:55.558+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:55.811 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:55.558+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:55.811 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:55.558+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:55.811 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:55.559+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:56.110 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:55.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:56.110 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:55.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:56.110 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:55.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:56.110 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:55.828+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:56.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:56.108+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:56.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:56.108+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:56.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:56.108+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:56.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:56.108+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:56.830 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:56.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:56.830 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:56.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:56.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:56.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:56.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:56.544+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:57.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:56.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:57.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:56.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:57.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:56.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:57.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:56.829+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:57.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:57.105+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:57.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:57.105+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:57.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:57.105+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:57.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:57.105+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:57.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:57.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:57.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:57.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:57.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:57.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:57.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:57.516+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:58.092+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:58.092+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:58.092+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:58.093+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:57.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:57.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:57.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:57.869+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:58.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:58.507+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:58.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:58.507+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:58.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:58.507+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:58.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:58.508+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:59.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:58.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:59.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:58.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:59.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:58.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:49:59.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:58.857+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:49:59.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:59.123+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:59.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:59.123+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:59.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:59.123+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:49:59.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:49:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:49:59.123+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:49:59.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:59.545+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:59.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:59.545+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:59.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:59.545+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:49:59.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:49:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:49:59.545+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:00.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:59.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:00.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:59.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:00.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:59.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:00.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:49:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:49:59.868+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:00.098+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:00.098+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:00.098+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:00.098+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:00.799 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:00.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:00.799 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:00.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:00.799 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:00.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:00.799 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:00.524+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:01.125 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:50:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:50:00.796+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 1119 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:50:01.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:00.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:01.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:00.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:01.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:00.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:01.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:00.869+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:01.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:01.123+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:01.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:01.123+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:01.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:01.123+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:01.124+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:01.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:01.498+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:01.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:01.498+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:01.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:01.498+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:01.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:01.499+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:02.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:01.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:02.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:01.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:02.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:01.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:02.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:01.857+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:02.079+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:02.079+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:02.079+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:02.080+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:02.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:02.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:02.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:02.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:02.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:02.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:02.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:02.473+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:02.878+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:02.878+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:02.878+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:02.878+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:03.036+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:03.036+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:03.036+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:03.036+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:03.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:03.518+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:03.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:03.518+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:03.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:03.518+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:03.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:03.519+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:04.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:04.029+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:04.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:04.029+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:04.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:04.029+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:04.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:04.029+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:04.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:03.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:04.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:03.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:04.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:03.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:04.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:03.880+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:04.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:04.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:04.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:04.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:04.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:04.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:04.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:04.542+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:05.007+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:05.007+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:05.007+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:05.008+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:04.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:04.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:04.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:04.887+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:05.799 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:05.496+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:05.800 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:05.496+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:05.800 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:05.496+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:05.800 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:05.497+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:06.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:50:05.797+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 1163 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:50:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:05.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:05.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:05.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:05.963+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:05.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:05.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:05.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:05.896+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:06.902 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:06.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:06.902 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:06.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:06.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:06.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:06.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:06.475+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:06.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:06.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:06.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:06.966+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:06.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:06.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:06.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:06.901+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:07.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:07.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:07.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:07.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:07.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:07.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:07.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:07.474+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:08.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:07.955+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:08.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:07.955+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:07.955+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:07.955+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:07.940+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:07.940+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:07.940+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:07.941+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:08.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:08.453+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:08.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:08.453+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:08.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:08.453+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:08.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:08.454+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:09.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:08.923+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:09.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:08.923+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:09.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:08.923+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:08.923+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:08.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:08.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:08.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:08.904+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:09.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:09.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:09.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:09.469+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:09.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:09.469+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:09.469+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:10.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:09.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:10.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:09.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:10.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:09.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:10.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:09.915+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:09.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:09.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:09.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:09.921+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:10.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:10.494+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:10.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:10.494+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:10.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:10.494+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:10.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:10.495+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:11.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:50:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:50:10.798+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 1208 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:50:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:10.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:10.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:10.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:10.957+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:10.908+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:10.908+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:10.908+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:10.908+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:11.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:11.459+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:11.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:11.459+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:11.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:11.459+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:11.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:11.460+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:11.926+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:11.926+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:11.926+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:11.927+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:11.996+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:11.996+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:11.996+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:11.997+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:12.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:12.420+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:12.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:12.420+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:12.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:12.420+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:12.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:12.421+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:12.920+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:12.920+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:12.920+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:12.921+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:13.013+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:13.013+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:13.013+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:13.013+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:13.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:13.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:13.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:13.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:13.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:13.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:13.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:13.441+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:13.923+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:13.923+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:13.923+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:13.924+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:14.014+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:14.014+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:14.014+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:14.015+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:14.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:14.433+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:14.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:14.433+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:14.433+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:14.434+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:15.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:14.950+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:15.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:14.950+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:15.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:14.950+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:14.950+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:15.038+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:15.038+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:15.038+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:15.039+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:15.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:15.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:15.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:15.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:15.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:15.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:15.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:15.397+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:16.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:50:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:50:15.798+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 1249 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:50:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:15.969+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:15.969+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:15.969+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:15.970+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:16.034+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:16.034+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:16.034+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:16.035+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:16.363+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:16.363+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:16.363+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:16.364+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:17.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:50:17.236+0000 7fa3f4bf1640 0 monclient(hunting): authenticate timed out after 300 2024-09-20T19:50:17.238 INFO:teuthology.orchestra.run.smithi114.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-09-20T19:50:17.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:17.012+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:17.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:17.012+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:17.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:17.012+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:17.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:17.013+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:17.063+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:17.063+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:17.063+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:17.063+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:17.596 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:17.356+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:17.596 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:17.356+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:17.596 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:17.356+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:17.596 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:17.357+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:18.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-09-20T19:50:18.329 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:50:18.347 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:18.035+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:18.347 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:18.035+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:18.347 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:18.035+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:18.347 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:18.036+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:18.347 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:18.015+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:18.347 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:18.015+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:18.347 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:18.015+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:18.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:18.015+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:18.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:18.345+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:18.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:18.345+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:18.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:18.345+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:18.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:18.346+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:19.386 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:19.040+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:19.386 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:19.040+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:19.387 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:19.040+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:19.387 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:19.041+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:19.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:19.062+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:19.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:19.062+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:19.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:19.062+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:19.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:19.063+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:19.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:19.384+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:19.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:19.384+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:19.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:19.384+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:19.385+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:20.416 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:20.073+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:20.417 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:20.073+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:20.417 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:20.073+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:20.417 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:20.074+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:20.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:20.020+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:20.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:20.020+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:20.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:20.020+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:20.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:20.021+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:20.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:20.414+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:20.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:20.414+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:20.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:20.414+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:20.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:20.415+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:21.053 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:50:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:50:20.799+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 1293 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:50:21.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:21.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:21.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:21.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:21.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:21.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:21.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:21.059+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:21.356 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:21.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:21.356 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:21.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:21.356 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:21.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:21.356 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:21.053+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:21.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:21.380+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:21.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:21.380+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:21.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:21.380+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:21.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:21.380+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:22.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:22.094+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:22.383 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:22.094+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:22.383 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:22.094+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:22.383 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:22.095+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:22.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:22.071+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:22.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:22.071+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:22.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:22.071+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:22.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:22.072+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:22.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:22.380+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:22.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:22.380+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:22.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:22.380+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:22.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:22.381+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:23.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:23.128+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:23.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:23.128+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:23.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:23.128+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:23.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:23.129+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:23.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:23.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:23.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:23.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:23.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:23.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:23.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:23.038+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:23.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:23.379+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:23.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:23.379+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:23.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:23.379+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:23.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:23.380+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:24.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:24.172+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:24.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:24.172+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:24.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:24.172+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:24.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:24.172+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:24.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:24.075+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:24.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:24.075+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:24.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:24.075+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:24.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:24.075+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:24.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:24.412+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:24.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:24.412+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:24.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:24.412+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:24.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:24.413+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:25.461 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:25.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:25.461 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:25.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:25.461 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:25.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:25.461 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:25.070+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:25.462 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:25.174+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:25.462 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:25.174+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:25.462 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:25.174+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:25.462 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:25.175+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:25.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:25.459+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:25.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:25.459+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:25.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:25.459+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:25.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:25.460+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:26.082 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:50:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:50:25.800+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 1335 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:50:26.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:26.080+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:26.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:26.080+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:26.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:26.080+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:26.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:26.081+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:26.437 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:26.153+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:26.437 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:26.153+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:26.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:26.153+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:26.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:26.153+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:26.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:26.435+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:26.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:26.435+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:26.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:26.435+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:26.436+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:27.398 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:27.107+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:27.398 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:27.107+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:27.398 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:27.107+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:27.398 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:27.108+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:27.398 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:27.045+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:27.398 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:27.045+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:27.398 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:27.045+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:27.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:27.046+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:27.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:27.396+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:27.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:27.396+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:27.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:27.396+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:27.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:27.396+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:28.440 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:28.139+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:28.440 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:28.139+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:28.440 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:28.139+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:28.440 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:28.140+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:28.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:28.013+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:28.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:28.013+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:28.441 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:28.013+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:28.441 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:28.014+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:28.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:28.438+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:28.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:28.438+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:28.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:28.438+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:28.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:28.439+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:29.464 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:29.185+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:29.464 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:29.185+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:29.464 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:29.185+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:29.464 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:29.186+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:29.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:29.014+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:29.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:29.014+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:29.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:29.014+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:29.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:29.015+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:29.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:29.462+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:29.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:29.462+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:29.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:29.462+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:29.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:29.462+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:30.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:29.993+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:30.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:29.993+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:30.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:29.993+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:29.993+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:30.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:30.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:30.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:30.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:30.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:30.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:30.442+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:30.234+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:30.234+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:30.234+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:30.234+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:31.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:50:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:50:30.800+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 1424 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:50:31.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:31.000+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:31.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:31.000+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:31.000+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:31.000+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:31.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:31.432+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:31.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:31.432+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:31.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:31.432+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:31.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:31.433+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:31.272+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:31.272+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:31.272+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:31.272+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:32.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:31.971+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:32.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:31.971+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:32.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:31.971+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:31.971+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:32.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:32.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:32.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:32.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:32.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:32.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:32.316+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:32.464+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:32.464+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:32.464+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:32.464+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:33.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:32.989+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:33.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:32.989+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:33.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:32.989+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:33.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:32.989+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:33.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:33.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:33.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:33.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:33.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:33.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:33.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:33.348+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:33.719 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:33.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:33.719 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:33.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:33.719 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:33.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:33.719 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:33.460+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:34.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:33.954+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:34.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:33.954+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:34.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:33.954+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:33.954+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:34.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:34.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:34.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:34.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:34.331+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:34.450+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:34.450+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:34.450+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:34.450+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:35.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:34.970+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:35.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:34.970+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:34.970+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:34.972+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:35.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:35.303+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:35.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:35.303+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:35.303+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:35.303+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:35.448+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:35.448+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:35.448+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:35.449+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:36.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:50:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:50:35.800+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 1515 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:50:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:35.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:35.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:35.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:35.933+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:36.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:36.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:36.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:36.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:36.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:36.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:36.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:36.343+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:36.491+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:36.491+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:36.491+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:36.492+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:36.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:36.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:37.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:36.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:37.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:36.954+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:37.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:37.316+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:37.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:37.316+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:37.316+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:37.317+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:37.512+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:37.512+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:37.512+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:37.512+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:38.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:37.917+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:38.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:37.917+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:38.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:37.917+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:38.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:37.918+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:38.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:38.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:38.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:38.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:38.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:38.298+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:38.517+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:38.517+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:38.517+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:38.517+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:39.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:38.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:39.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:38.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:39.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:38.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:39.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:38.887+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:39.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:39.520+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:39.520+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:39.520+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:39.521+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:39.302+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:39.302+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:39.302+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:39.302+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:40.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:39.841+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:40.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:39.841+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:39.841+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:39.841+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:40.557 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:40.288+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:40.557 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:40.288+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:40.557 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:40.288+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:40.557 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:40.289+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:40.812 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:50:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:50:40.801+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 1609 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:50:40.812 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:40.555+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:40.812 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:40.555+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:40.812 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:40.555+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:40.812 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:40.555+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:41.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:40.854+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:41.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:40.854+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:40.854+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:40.855+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:41.571 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:41.325+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:41.571 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:41.325+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:41.571 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:41.325+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:41.571 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:41.326+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:41.901 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:41.569+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:41.901 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:41.569+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:41.901 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:41.569+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:41.901 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:41.570+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:42.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:41.898+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:41.898+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:41.898+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:41.899+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:42.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:42.325+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:42.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:42.325+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:42.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:42.325+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:42.326+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:42.542+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:42.542+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:42.542+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:42.543+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:43.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:42.874+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:43.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:42.874+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:43.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:42.874+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:42.875+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:43.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:43.368+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:43.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:43.368+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:43.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:43.368+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:43.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:43.369+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:43.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:43.585+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:43.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:43.585+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:43.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:43.585+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:43.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:43.585+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:44.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:43.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:44.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:43.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:44.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:43.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:44.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:43.843+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:44.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:44.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:44.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:44.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:44.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:44.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:44.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:44.374+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:44.608+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:44.608+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:44.608+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:44.609+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:45.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:44.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:45.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:44.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:45.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:44.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:44.882+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:45.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:45.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:45.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:45.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:45.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:45.604+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:45.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:45.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:45.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:45.385+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:50:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:50:45.802+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 1701 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:50:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:45.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:45.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:45.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:45.834+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:46.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:46.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:46.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:46.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:46.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:46.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:46.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:46.345+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:46.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:46.626+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:46.626+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:46.626+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:46.626+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:46.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:46.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:46.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:46.791+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:47.615 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:47.361+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:47.615 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:47.361+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:47.615 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:47.361+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:47.615 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:47.362+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:47.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:47.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:47.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:47.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:47.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:47.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:47.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:47.834+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:47.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:47.613+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:47.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:47.613+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:47.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:47.613+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:47.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:47.614+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:48.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:48.316+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:48.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:48.316+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:48.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:48.316+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:48.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:48.317+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:48.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:48.841+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:48.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:48.841+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:48.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:48.841+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:48.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:48.841+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:48.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:48.630+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:48.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:48.630+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:48.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:48.630+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:48.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:48.631+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:49.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:49.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:49.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:49.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:49.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:49.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:49.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:49.336+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:49.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:49.810+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:49.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:49.810+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:49.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:49.810+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:49.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:49.810+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:49.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:49.634+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:49.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:49.634+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:49.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:49.634+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:49.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:49.635+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:50.632 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:50.372+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:50.632 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:50.372+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:50.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:50.372+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:50.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:50.372+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:50.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:50:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:50:50.803+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 1797 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:50:50.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:50.630+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:50.630+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:50.630+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:50.631+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:50.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:50.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:50.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:50.795+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:51.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:51.422+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:51.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:51.422+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:51.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:51.422+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:51.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:51.422+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:51.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:51.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:51.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:51.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:51.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:51.752+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:51.586+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:51.586+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:51.586+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:51.586+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:52.800 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:52.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:52.801 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:52.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:52.801 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:52.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:52.801 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:52.544+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:52.801 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:52.465+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:52.801 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:52.465+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:52.801 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:52.465+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:52.801 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:52.466+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:52.798+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:52.798+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:52.798+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:52.799+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:53.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:53.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:53.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:53.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:53.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:53.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:53.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:53.419+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:53.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:53.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:53.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:53.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:53.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:53.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:53.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:53.505+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:53.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:53.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:53.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:53.753+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:54.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:54.436+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:54.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:54.436+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:54.436+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:54.437+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:54.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:54.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:54.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:54.532+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:55.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:54.781+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:55.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:54.781+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:55.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:54.781+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:55.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:54.782+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:55.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:55.409+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:55.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:55.409+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:55.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:55.409+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:55.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:55.409+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:55.806 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:55.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:55.806 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:55.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:55.806 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:55.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:55.806 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:55.523+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:56.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:50:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:50:55.804+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 1889 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:50:56.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:55.822+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:56.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:55.822+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:56.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:55.822+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:55.822+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:56.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:56.520+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:56.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:56.520+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:56.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:56.520+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:56.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:56.521+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:56.403+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:56.403+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:56.403+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:56.404+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:57.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:56.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:57.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:56.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:57.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:56.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:56.867+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:57.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:57.427+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:57.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:57.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:57.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:57.428+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:57.474+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:57.474+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:57.474+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:57.475+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:57.902+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:57.902+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:57.902+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:57.902+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:58.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:58.455+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:58.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:58.455+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:58.455+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:58.456+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:50:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:58.445+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:58.445+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:58.445+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:58.446+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:59.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:58.945+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:59.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:58.945+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:59.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:58.945+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:50:59.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:58.945+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:59.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:59.427+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:59.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:59.427+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:59.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:59.427+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:50:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:50:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:50:59.427+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:50:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:59.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:59.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:59.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:50:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:50:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:50:59.472+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:00.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:59.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:00.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:59.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:59.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:50:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:50:59.958+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:00.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:00.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:00.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:00.465+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:00.444+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:00.444+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:00.444+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:00.444+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:01.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:51:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:51:00.804+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 1984 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:51:01.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:00.975+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:00.975+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:00.975+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:00.976+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:01.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:01.454+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:01.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:01.454+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:01.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:01.454+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:01.455+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:01.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:01.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:01.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:01.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:01.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:01.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:01.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:01.471+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:02.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:01.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:02.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:01.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:02.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:01.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:02.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:01.933+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:02.918 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:02.479+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:02.918 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:02.479+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:02.918 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:02.479+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:02.918 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:02.480+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:02.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:02.495+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:02.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:02.495+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:02.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:02.495+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:02.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:02.496+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:02.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:02.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:02.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:02.916+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:03.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:03.508+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:03.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:03.508+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:03.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:03.508+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:03.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:03.508+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:03.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:03.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:03.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:03.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:03.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:03.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:03.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:03.491+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:03.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:03.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:03.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:03.934+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:04.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:04.475+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:04.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:04.475+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:04.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:04.475+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:04.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:04.476+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:04.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:04.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:04.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:04.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:04.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:04.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:04.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:04.491+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:05.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:04.958+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:05.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:04.958+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:05.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:04.958+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:05.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:04.958+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:05.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:05.461+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:05.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:05.461+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:05.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:05.461+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:05.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:05.461+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:05.807 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:05.501+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:05.807 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:05.501+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:05.807 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:05.501+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:05.808 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:05.501+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:06.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:51:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:51:05.805+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 2075 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:51:06.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:05.940+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:06.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:05.940+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:05.940+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:05.941+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:06.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:06.484+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:06.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:06.484+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:06.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:06.484+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:06.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:06.485+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:06.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:06.534+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:06.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:06.534+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:06.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:06.534+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:06.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:06.534+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:07.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:06.943+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:07.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:06.943+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:07.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:06.943+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:07.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:06.944+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:07.903 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:07.532+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:07.903 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:07.532+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:07.903 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:07.532+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:07.903 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:07.533+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:07.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:07.483+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:07.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:07.483+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:07.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:07.483+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:07.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:07.484+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:08.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:07.901+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:08.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:07.901+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:08.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:07.901+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:08.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:07.902+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:08.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:08.573+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:08.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:08.573+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:08.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:08.573+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:08.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:08.574+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:08.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:08.485+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:08.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:08.485+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:08.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:08.485+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:08.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:08.486+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:09.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:08.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:09.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:08.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:09.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:08.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:09.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:08.871+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:09.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:09.594+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:09.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:09.594+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:09.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:09.594+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:09.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:09.594+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:09.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:09.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:09.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:09.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:09.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:09.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:09.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:09.473+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:10.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:09.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:10.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:09.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:10.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:09.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:10.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:09.908+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:10.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:10.576+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:10.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:10.576+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:10.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:10.576+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:10.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:10.577+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:10.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:10.496+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:10.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:10.496+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:10.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:10.496+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:10.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:10.497+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:11.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:51:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:51:10.806+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 2170 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:51:11.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:10.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:11.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:10.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:10.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:10.922+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:11.607 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:11.457+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:11.607 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:11.457+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:11.607 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:11.457+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:11.607 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:11.458+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:11.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:11.605+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:11.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:11.605+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:11.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:11.605+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:11.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:11.606+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:12.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:11.965+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:12.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:11.965+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:12.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:11.965+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:12.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:11.966+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:12.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:12.654+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:12.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:12.654+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:12.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:12.654+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:12.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:12.654+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:12.453+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:12.453+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:12.453+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:12.453+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:13.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:12.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:13.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:12.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:13.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:12.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:13.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:12.932+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:13.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:13.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:13.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:13.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:13.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:13.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:13.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:13.673+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:13.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:13.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:13.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:13.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:13.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:13.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:13.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:13.479+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:14.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:13.898+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:14.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:13.898+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:14.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:13.898+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:14.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:13.898+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:14.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:14.711+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:14.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:14.711+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:14.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:14.711+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:14.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:14.711+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:14.833 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:14.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:14.833 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:14.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:14.833 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:14.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:14.834 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:14.491+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:15.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:14.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:15.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:14.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:15.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:14.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:15.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:14.881+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:15.809 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:15.744+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:15.809 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:15.744+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:15.809 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:15.744+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:15.809 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:15.745+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:15.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:15.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:15.810 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:15.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:15.810 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:15.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:15.810 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:15.533+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:16.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:51:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:51:15.807+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 2261 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:51:16.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:15.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:16.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:15.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:16.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:15.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:16.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:15.876+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:16.928 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:16.731+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:16.928 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:16.731+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:16.928 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:16.731+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:16.928 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:16.732+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:16.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:16.544+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:16.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:16.544+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:16.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:16.544+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:16.929 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:16.545+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:17.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:16.926+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:17.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:16.926+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:17.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:16.926+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:17.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:16.927+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:17.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:17.539+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:17.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:17.539+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:17.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:17.539+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:17.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:17.539+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:17.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:17.698+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:17.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:17.698+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:17.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:17.698+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:17.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:17.698+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:18.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:17.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:18.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:17.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:18.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:17.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:18.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:17.967+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:18.930 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:18.714+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:18.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:18.714+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:18.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:18.714+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:18.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:18.714+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:18.931 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:18.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:18.931 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:18.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:18.931 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:18.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:18.931 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:18.565+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:19.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:18.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:19.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:18.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:19.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:18.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:19.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:18.929+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:19.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:19.665+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:19.665+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:19.665+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:19.665+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:19.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:19.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:19.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:19.532+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:20.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:19.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:20.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:19.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:20.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:19.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:19.967+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:20.809 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:20.635+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:20.810 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:20.635+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:20.810 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:20.635+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:20.810 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:20.636+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:20.810 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:20.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:20.810 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:20.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:20.811 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:20.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:20.811 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:20.526+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:21.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:51:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:51:20.806+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 2355 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:51:21.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:20.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:21.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:20.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:20.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:20.940+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:21.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:21.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:21.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:21.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:21.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:21.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:21.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:21.515+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:21.933 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:21.674+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:21.933 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:21.674+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:21.933 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:21.674+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:21.933 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:21.675+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:22.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:21.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:22.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:21.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:22.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:21.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:21.930+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:22.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:22.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:22.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:22.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:22.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:22.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:22.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:22.475+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:22.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:22.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:22.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:22.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:22.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:22.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:22.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:22.668+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:23.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:22.917+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:23.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:22.917+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:23.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:22.917+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:23.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:22.918+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:23.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:23.500+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:23.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:23.500+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:23.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:23.500+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:23.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:23.500+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:23.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:23.695+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:23.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:23.695+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:23.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:23.695+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:23.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:23.695+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:24.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:23.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:24.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:23.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:24.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:23.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:24.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:23.959+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:24.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:24.690+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:24.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:24.690+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:24.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:24.690+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:24.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:24.690+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:24.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:24.524+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:24.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:24.524+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:24.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:24.524+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:24.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:24.525+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:25.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:24.937+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:25.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:24.937+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:25.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:24.937+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:24.938+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:25.811 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:25.684+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:25.811 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:25.684+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:25.811 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:25.684+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:25.811 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:25.684+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:25.811 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:25.556+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:25.811 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:25.556+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:25.812 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:25.556+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:25.812 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:25.556+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:26.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:51:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:51:25.809+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 2447 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:51:26.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:25.944+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:26.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:25.944+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:25.944+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:25.945+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:26.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:26.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:26.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:26.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:26.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:26.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:26.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:26.720+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:26.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:26.534+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:26.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:26.534+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:26.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:26.534+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:26.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:26.535+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:27.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:26.961+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:27.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:26.961+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:27.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:26.961+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:27.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:26.961+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:27.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:27.724+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:27.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:27.724+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:27.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:27.724+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:27.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:27.725+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:27.584+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:27.584+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:27.584+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:27.585+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:28.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:27.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:28.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:27.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:28.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:27.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:28.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:27.980+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:28.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:28.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:28.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:28.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:28.747+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:28.562+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:28.562+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:28.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:28.563+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:29.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:28.998+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:29.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:28.998+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:29.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:28.998+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:29.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:28.999+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:29.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:29.697+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:29.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:29.697+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:29.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:29.697+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:29.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:29.698+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:29.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:29.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:29.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:29.525+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:30.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:29.973+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:30.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:29.973+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:30.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:29.973+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:30.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:29.974+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:30.812 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:30.720+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:30.812 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:30.720+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:30.812 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:30.720+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:30.812 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:30.720+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:30.812 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:30.524+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:30.813 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:30.524+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:30.813 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:30.524+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:30.813 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:30.525+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:31.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:51:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:51:30.810+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 2541 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:51:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:30.942+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:30.942+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:30.942+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:30.943+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:31.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:31.710+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:31.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:31.710+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:31.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:31.710+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:31.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:31.710+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:31.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:31.569+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:31.569+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:31.569+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:32.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:31.972+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:32.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:31.972+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:32.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:31.972+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:32.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:31.972+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:32.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:32.739+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:32.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:32.739+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:32.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:32.739+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:32.739+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:32.557+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:32.557+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:32.557+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:32.558+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:33.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:32.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:33.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:32.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:33.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:32.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:33.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:32.982+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:33.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:33.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:33.784 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:33.602+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:33.784 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:33.602+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:33.785 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:33.602+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:33.785 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:33.602+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:34.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:33.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:34.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:33.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:34.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:33.783+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:33.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:33.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:33.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:33.940+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:34.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:34.778+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:34.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:34.778+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:34.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:34.778+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:34.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:34.779+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:34.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:34.593+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:34.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:34.593+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:34.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:34.593+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:34.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:34.594+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:35.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:34.908+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:34.908+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:34.908+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:34.909+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:35.909 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:51:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:51:35.810+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 2633 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:51:35.909 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:35.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:35.909 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:35.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:35.909 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:35.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:35.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:35.613+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:35.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:35.738+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:35.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:35.738+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:35.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:35.738+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:35.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:35.739+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:36.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:35.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:36.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:35.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:36.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:35.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:36.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:35.907+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:36.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:36.594+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:36.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:36.594+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:36.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:36.594+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:36.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:36.595+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:36.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:36.777+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:36.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:36.777+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:36.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:36.777+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:36.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:36.778+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:36.942+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:36.942+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:36.942+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:36.942+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:37.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:37.798+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:37.927 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:37.798+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:37.927 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:37.798+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:37.927 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:37.799+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:37.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:37.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:37.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:37.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:37.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:37.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:37.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:37.565+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:38.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:37.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:38.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:37.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:38.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:37.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:38.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:37.925+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:38.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:38.517+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:38.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:38.517+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:38.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:38.517+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:38.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:38.518+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:38.786+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:38.786+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:38.786+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:38.787+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:38.893+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:38.893+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:38.893+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:38.894+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:39.813 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:39.560+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:39.814 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:39.560+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:39.814 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:39.560+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:39.814 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:39.561+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:40.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:39.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:40.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:39.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:40.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:39.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:39.812+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:39.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:39.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:39.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:39.891+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:40.813 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:40.788+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:40.813 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:40.788+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:40.814 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:40.788+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:40.814 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:40.789+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:40.814 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:40.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:40.814 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:40.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:40.814 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:40.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:40.814 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:40.551+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:51:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:51:40.811+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 2727 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:51:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:40.851+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:40.851+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:40.851+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:40.852+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:41.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:41.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:41.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:41.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:41.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:41.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:41.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:41.565+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:41.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:41.806+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:41.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:41.806+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:41.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:41.806+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:41.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:41.807+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:42.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:41.829+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:42.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:41.829+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:42.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:41.829+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:41.830+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:42.844 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:42.789+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:42.844 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:42.789+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:42.844 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:42.789+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:42.844 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:42.790+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:42.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:42.588+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:42.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:42.588+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:42.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:42.588+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:42.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:42.589+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:43.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:42.842+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:43.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:42.842+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:43.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:42.842+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:42.843+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:43.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:43.566+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:43.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:43.566+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:43.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:43.566+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:43.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:43.567+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:44.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:43.891+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:44.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:43.891+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:44.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:43.891+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:44.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:43.891+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:43.829+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:43.829+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:43.829+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:43.830+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:44.846 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:44.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:44.846 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:44.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:44.846 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:44.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:44.847 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:44.568+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:45.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:44.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:45.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:44.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:45.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:44.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:45.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:44.845+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:44.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:44.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:44.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:44.865+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:45.814 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:45.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:45.814 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:45.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:45.814 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:45.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:45.815 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:45.799+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:45.815 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:45.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:45.815 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:45.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:45.815 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:45.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:45.815 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:45.579+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:46.140 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:51:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:51:45.812+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 2819 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:51:46.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:45.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:46.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:45.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:46.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:45.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:46.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:45.845+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:46.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:46.562+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:46.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:46.562+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:46.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:46.562+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:46.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:46.563+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:47.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:46.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:47.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:46.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:46.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:46.841+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:46.860+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:46.860+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:46.860+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:46.861+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:47.828 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:47.574+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:47.828 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:47.574+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:47.828 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:47.574+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:47.828 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:47.575+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:48.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:47.826+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:48.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:47.826+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:47.826+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:47.826+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:47.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:47.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:47.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:47.873+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:48.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:48.781+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:48.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:48.781+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:48.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:48.781+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:48.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:48.782+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:48.841 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:48.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:48.841 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:48.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:48.841 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:48.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:48.841 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:48.533+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:49.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:48.838+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:49.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:48.838+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:49.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:48.838+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:48.839+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:49.829 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:49.504+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:49.829 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:49.504+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:49.829 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:49.504+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:49.829 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:49.505+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:49.826+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:49.826+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:49.826+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:49.827+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:50.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:49.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:50.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:49.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:50.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:49.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:50.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:49.836+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:50.802 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:50.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:50.802 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:50.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:50.802 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:50.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:50.802 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:50.505+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:51.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:51:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:51:50.813+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 2915 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:51:51.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:50.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:51.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:50.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:50.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:50.800+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:50.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:50.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:50.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:50.838+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:51.813 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:51.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:51.813 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:51.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:51.814 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:51.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:51.814 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:51.515+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:52.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:51.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:52.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:51.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:51.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:51.810+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:51.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:51.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:51.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:51.843+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:52.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:52.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:52.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:52.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:52.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:52.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:52.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:52.506+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:53.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:52.788+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:53.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:52.788+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:53.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:52.788+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:52.789+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:52.858+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:52.858+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:52.858+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:52.858+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:53.812 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:53.552+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:53.812 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:53.552+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:53.812 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:53.552+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:53.813 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:53.553+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:53.877+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:53.877+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:53.877+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:53.878+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:54.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:53.810+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:54.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:53.810+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:54.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:53.810+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:54.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:53.811+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:54.854 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:54.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:54.854 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:54.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:54.855 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:54.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:54.855 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:54.555+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:54.852+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:54.852+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:54.852+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:54.853+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:54.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:54.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:54.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:54.859+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:55.816 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:55.508+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:55.816 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:55.508+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:55.816 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:55.508+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:55.816 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:55.508+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:56.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:51:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:51:55.814+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 3007 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:51:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:55.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:55.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:55.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:55.908+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:55.897+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:55.898+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:55.898+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:55.898+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:56.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:56.508+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:56.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:56.508+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:56.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:56.508+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:56.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:56.508+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:56.889+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:56.889+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:56.889+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:56.889+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:56.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:56.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:56.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:56.934+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:57.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:57.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:57.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:57.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:57.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:57.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:57.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:57.542+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:57.903+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:57.903+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:57.903+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:57.903+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:57.931+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:57.931+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:57.931+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:57.931+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:58.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:58.586+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:58.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:58.586+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:58.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:58.586+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:58.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:58.586+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:59.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:58.946+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:59.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:58.946+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:59.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:58.946+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:51:59.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:58.946+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:51:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:58.862+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:58.862+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:58.862+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:51:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:58.863+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:51:59.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:59.582+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:59.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:59.582+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:59.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:59.582+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:51:59.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:51:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:51:59.583+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:59.922+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:59.922+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:59.922+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:51:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:51:59.923+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:59.882+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:59.882+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:59.882+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:51:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:51:59.882+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:00.924 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:52:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:52:00.814+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 3102 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:52:00.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:00.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:00.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:00.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:00.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:00.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:00.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:00.610+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:01.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:00.922+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:01.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:00.922+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:01.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:00.922+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:00.923+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:00.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:00.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:00.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:00.933+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:01.931 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:01.638+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:01.931 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:01.638+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:01.931 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:01.638+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:01.931 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:01.639+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:02.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:01.929+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:02.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:01.929+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:02.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:01.929+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:01.930+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:01.955+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:01.955+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:01.955+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:01.956+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:02.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:02.596+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:02.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:02.596+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:02.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:02.596+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:02.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:02.597+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:02.967+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:02.967+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:02.967+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:02.968+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:02.943+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:02.943+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:02.943+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:02.944+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:03.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:03.586+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:03.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:03.586+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:03.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:03.586+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:03.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:03.587+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:03.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:03.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:03.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:03.939+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:03.917+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:03.917+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:03.917+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:03.918+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:04.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:04.567+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:04.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:04.567+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:04.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:04.567+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:04.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:04.568+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:04.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:04.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:04.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:04.915+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:04.875+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:04.875+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:04.875+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:04.875+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:05.869 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:52:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:52:05.814+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 3195 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:52:05.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:05.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:05.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:05.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:05.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:05.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:05.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:05.579+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:06.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:05.876+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:06.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:05.876+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:06.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:05.876+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:06.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:05.876+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:05.901+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:05.901+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:05.901+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:05.902+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:06.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:06.601+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:06.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:06.601+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:06.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:06.601+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:06.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:06.602+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:06.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:06.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:06.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:06.841+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:06.937+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:06.937+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:06.937+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:06.938+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:07.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:07.560+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:07.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:07.560+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:07.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:07.560+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:07.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:07.561+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:08.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:07.854+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:08.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:07.854+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:08.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:07.854+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:08.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:07.855+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:07.947+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:07.947+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:07.947+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:07.947+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:08.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:08.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:08.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:08.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:08.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:08.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:08.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:08.526+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:09.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:08.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:09.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:08.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:09.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:08.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:09.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:08.982+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:08.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:08.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:08.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:08.842+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:09.821 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:09.513+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:09.822 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:09.513+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:09.822 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:09.513+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:09.822 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:09.513+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:10.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:09.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:10.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:09.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:10.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:09.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:10.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:09.819+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:10.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:09.994+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:10.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:09.994+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:10.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:09.994+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:10.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:09.995+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:10.818 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:10.520+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:10.818 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:10.520+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:10.818 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:10.520+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:10.818 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:10.521+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:11.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:52:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:52:10.816+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 3292 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:52:11.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:10.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:10.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:10.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:10.849+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:10.994+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:10.994+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:10.994+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:10.995+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:11.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:11.530+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:11.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:11.530+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:11.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:11.530+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:11.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:11.531+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:11.892+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:11.892+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:11.892+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:11.893+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:12.038+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:12.038+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:12.038+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:12.039+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:12.493 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:12.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:12.494 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:12.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:12.494 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:12.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:12.494 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:12.491+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:13.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:12.847+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:13.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:12.847+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:13.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:12.847+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:13.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:12.848+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:13.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:13.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:13.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:13.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:13.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:13.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:13.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:13.085+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:13.850 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:13.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:13.850 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:13.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:13.850 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:13.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:13.850 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:13.490+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:13.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:13.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:13.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:13.849+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:14.065+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:14.065+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:14.065+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:14.066+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:14.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:14.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:14.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:14.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:14.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:14.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:14.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:14.526+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:15.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:14.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:14.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:14.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:14.879+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:15.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:15.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:15.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:15.085+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:15.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:15.499+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:15.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:15.499+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:15.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:15.499+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:15.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:15.500+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:16.117 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:52:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:52:15.817+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 3385 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:52:16.118 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:15.926+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:16.118 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:15.926+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:16.118 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:15.926+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:16.118 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:15.927+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:16.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:16.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:16.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:16.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:16.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:16.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:16.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:16.116+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:16.916 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:16.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:16.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:16.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:16.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:16.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:16.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:16.489+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:17.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:16.914+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:17.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:16.914+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:16.914+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:16.915+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:17.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:17.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:17.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:17.070+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:17.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:17.509+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:17.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:17.509+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:17.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:17.509+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:17.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:17.509+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:17.917+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:17.917+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:17.917+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:17.918+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:18.047+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:18.047+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:18.047+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:18.047+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:18.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:18.528+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:18.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:18.528+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:18.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:18.528+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:18.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:18.529+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:19.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:18.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:19.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:18.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:18.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:18.881+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:19.083+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:19.083+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:19.083+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:19.084+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:19.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:19.556+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:19.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:19.556+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:19.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:19.556+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:19.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:19.557+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:20.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:19.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:20.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:19.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:20.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:19.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:20.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:19.840+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:20.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:20.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:20.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:20.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:20.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:20.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:20.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:20.121+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:20.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:20.511+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:20.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:20.511+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:20.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:20.511+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:20.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:20.512+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:21.137 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:52:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:52:20.818+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 3480 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:52:21.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:20.817+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:21.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:20.817+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:21.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:20.817+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:21.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:20.817+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:21.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:21.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:21.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:21.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:21.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:21.136+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:21.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:21.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:21.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:21.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:21.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:21.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:21.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:21.554+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:22.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:21.863+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:22.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:21.863+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:22.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:21.863+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:22.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:21.864+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:22.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:22.142+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:22.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:22.142+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:22.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:22.142+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:22.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:22.142+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:22.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:22.535+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:22.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:22.535+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:22.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:22.535+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:22.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:22.536+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:23.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:22.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:23.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:22.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:23.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:22.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:23.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:22.859+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:23.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:23.155+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:23.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:23.155+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:23.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:23.155+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:23.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:23.156+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:23.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:23.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:23.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:23.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:23.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:23.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:23.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:23.511+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:24.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:23.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:24.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:23.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:24.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:23.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:24.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:23.888+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:24.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:24.153+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:24.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:24.153+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:24.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:24.153+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:24.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:24.154+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:24.921 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:24.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:24.921 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:24.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:24.921 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:24.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:24.921 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:24.482+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:25.197 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:24.919+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:25.197 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:24.919+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:25.198 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:24.919+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:25.198 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:24.920+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:25.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:25.195+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:25.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:25.195+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:25.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:25.195+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:25.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:25.196+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:25.821 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:25.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:25.821 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:25.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:25.821 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:25.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:25.821 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:25.469+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:26.185 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:52:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:52:25.818+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 3573 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:52:26.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:25.923+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:26.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:25.923+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:26.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:25.923+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:26.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:25.924+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:26.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:26.183+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:26.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:26.183+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:26.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:26.183+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:26.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:26.184+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:26.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:26.449+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:26.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:26.449+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:26.449+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:26.449+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:27.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:26.952+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:27.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:26.952+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:27.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:26.952+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:27.195 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:26.953+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:27.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:27.404+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:27.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:27.404+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:27.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:27.404+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:27.405+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:27.192+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:27.192+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:27.192+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:27.193+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:28.437 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:27.987+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:28.437 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:27.987+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:28.437 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:27.987+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:28.437 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:27.988+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:28.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:28.211+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:28.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:28.211+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:28.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:28.211+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:28.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:28.211+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:28.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:28.435+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:28.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:28.435+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:28.435+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:28.436+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:29.433 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:28.972+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:29.433 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:28.972+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:29.433 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:28.972+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:29.433 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:28.972+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:29.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:29.181+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:29.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:29.181+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:29.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:29.181+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:29.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:29.182+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:29.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:29.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:29.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:29.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:29.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:29.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:29.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:29.431+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:29.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:29.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:29.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:29.939+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:30.171+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:30.171+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:30.171+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:30.171+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:30.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:30.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:30.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:30.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:30.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:30.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:30.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:30.481+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:31.212 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:52:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:52:30.819+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 3669 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:52:31.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:30.982+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:31.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:30.982+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:31.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:30.982+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:31.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:30.983+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:31.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:31.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:31.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:31.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:31.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:31.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:31.438+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:31.210+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:31.210+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:31.210+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:31.211+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:32.203 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:32.020+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:32.203 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:32.020+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:32.203 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:32.020+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:32.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:32.021+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:32.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:32.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:32.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:32.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:32.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:32.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:32.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:32.398+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:32.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:32.201+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:32.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:32.201+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:32.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:32.201+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:32.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:32.202+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:33.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:33.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:33.447 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:33.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:33.447 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:33.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:33.447 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:33.007+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:33.447 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:33.227+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:33.447 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:33.227+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:33.447 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:33.227+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:33.447 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:33.228+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:33.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:33.444+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:33.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:33.444+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:33.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:33.444+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:33.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:33.445+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:34.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:34.040+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:34.431 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:34.040+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:34.431 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:34.040+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:34.431 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:34.040+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:34.431 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:34.212+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:34.431 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:34.212+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:34.431 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:34.212+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:34.431 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:34.212+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:34.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:34.428+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:34.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:34.428+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:34.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:34.428+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:34.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:34.429+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:35.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:35.227+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:35.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:35.227+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:35.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:35.227+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:35.228+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:35.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:35.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:35.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:35.064+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:35.822 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:35.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:35.822 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:35.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:35.822 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:35.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:35.823 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:35.473+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:36.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:52:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:52:35.820+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 3761 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:52:36.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:36.036+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:36.036+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:36.036+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:36.037+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:36.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:36.246+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:36.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:36.246+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:36.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:36.246+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:36.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:36.247+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:36.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:36.497+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:36.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:36.497+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:36.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:36.497+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:36.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:36.498+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:37.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:37.262+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:37.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:37.262+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:37.262+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:37.263+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:37.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:37.041+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:37.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:37.041+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:37.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:37.041+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:37.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:37.041+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:37.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:37.526+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:37.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:37.526+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:37.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:37.526+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:37.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:37.527+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:38.070+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:38.070+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:38.070+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:38.071+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:38.288+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:38.288+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:38.288+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:38.288+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:38.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:38.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:38.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:38.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:38.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:38.574+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:39.338 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:39.028+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:39.338 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:39.028+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:39.339 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:39.028+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:39.339 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:39.028+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:39.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:39.537+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:39.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:39.537+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:39.537+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:39.538+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:39.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:39.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:39.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:39.337+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:40.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:40.011+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:40.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:40.011+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:40.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:40.011+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:40.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:40.012+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:40.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:40.583+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:40.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:40.583+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:40.583+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:40.584+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:40.350+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:40.350+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:40.350+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:40.351+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:52:40.821+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 3856 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:52:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:40.983+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:40.983+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:40.983+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:40.984+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:41.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:41.319+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:41.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:41.319+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:41.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:41.319+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:41.601 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:41.320+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:41.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:41.598+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:41.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:41.598+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:41.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:41.598+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:41.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:41.599+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:42.370 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:41.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:42.370 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:41.989+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:42.370 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:41.989+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:42.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:41.989+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:42.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:42.368+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:42.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:42.368+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:42.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:42.368+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:42.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:42.369+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:42.611+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:42.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:42.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:42.612+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:43.331 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:43.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:43.332 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:43.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:43.332 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:43.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:43.332 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:43.026+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:43.602 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:43.329+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:43.602 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:43.329+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:43.602 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:43.329+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:43.602 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:43.330+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:43.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:43.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:43.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:43.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:43.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:43.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:43.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:43.600+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:44.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:44.035+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:44.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:44.035+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:44.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:44.035+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:44.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:44.036+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:44.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:44.284+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:44.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:44.284+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:44.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:44.284+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:44.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:44.285+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:44.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:44.646+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:44.646+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:44.646+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:44.647+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:45.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:45.036+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:45.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:45.036+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:45.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:45.036+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:45.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:45.037+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:45.608 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:45.332+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:45.608 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:45.332+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:45.608 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:45.332+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:45.608 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:45.332+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:45.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:52:45.822+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 3949 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:52:45.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:45.605+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:45.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:45.605+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:45.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:45.606+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:45.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:45.606+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:45.990+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:45.990+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:45.990+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:45.990+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:46.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:46.284+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:46.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:46.284+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:46.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:46.284+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:46.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:46.285+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:46.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:46.640+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:46.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:46.640+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:46.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:46.640+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:46.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:46.641+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:47.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:46.952+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:46.952+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:46.952+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:46.953+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:47.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:47.311+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:47.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:47.311+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:47.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:47.311+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:47.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:47.313+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:47.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:47.640+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:47.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:47.640+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:47.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:47.640+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:47.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:47.641+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:48.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:47.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:48.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:47.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:47.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:47.931+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:48.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:48.290+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:48.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:48.290+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:48.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:48.290+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:48.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:48.290+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:48.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:48.669+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:48.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:48.669+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:48.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:48.669+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:48.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:48.670+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:49.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:48.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:49.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:48.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:49.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:48.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:49.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:48.933+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:49.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:49.251+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:49.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:49.251+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:49.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:49.251+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:49.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:49.251+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:49.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:49.672+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:49.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:49.672+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:49.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:49.672+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:49.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:49.673+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:49.942+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:49.942+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:49.942+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:49.942+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:50.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:50.250+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:50.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:50.250+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:50.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:50.250+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:50.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:50.251+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:50.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:52:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:52:50.823+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 4045 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:52:50.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:50.648+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:50.648+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:50.648+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:50.649+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:51.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:51.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:51.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:51.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:51.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:51.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:51.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:51.233+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:51.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:50.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:51.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:50.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:51.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:50.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:51.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:50.986+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:51.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:51.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:51.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:51.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:51.672+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:52.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:52.257+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:52.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:52.257+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:52.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:52.257+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:52.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:52.258+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:52.317 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:52.010+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:52.317 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:52.010+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:52.317 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:52.010+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:52.317 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:52.011+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:53.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:52.718+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:53.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:52.718+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:53.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:52.718+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:53.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:52.719+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:53.309 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:53.010+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:53.310 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:53.010+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:53.310 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:53.010+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:53.310 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:53.011+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:53.693 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:53.306+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:53.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:53.306+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:53.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:53.306+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:53.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:53.307+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:53.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:53.690+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:53.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:53.690+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:53.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:53.690+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:53.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:53.691+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:54.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:53.970+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:54.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:53.970+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:54.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:53.970+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:54.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:53.970+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:54.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:54.296+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:54.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:54.296+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:54.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:54.296+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:54.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:54.297+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:54.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:54.655+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:54.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:54.655+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:54.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:54.655+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:54.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:54.656+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:55.331 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:54.987+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:55.331 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:54.987+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:55.331 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:54.987+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:55.331 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:54.988+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:55.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:55.328+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:55.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:55.328+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:55.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:55.328+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:55.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:55.328+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:55.957 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:52:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:52:55.824+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 4136 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:52:55.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:55.634+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:55.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:55.634+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:55.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:55.634+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:55.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:55.634+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:56.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:55.954+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:56.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:55.954+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:56.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:55.954+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:56.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:55.955+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:56.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:56.355+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:56.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:56.355+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:56.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:56.355+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:56.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:56.356+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:56.937 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:56.635+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:56.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:56.635+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:56.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:56.635+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:56.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:56.636+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:56.935+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:56.935+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:56.935+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:56.936+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:57.617 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:57.317+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:57.617 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:57.317+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:57.617 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:57.317+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:57.617 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:57.318+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:57.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:57.615+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:57.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:57.615+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:57.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:57.615+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:57.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:57.616+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:57.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:57.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:57.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:57.916+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:58.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:58.289+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:58.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:58.289+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:58.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:58.289+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:58.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:58.290+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:58.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:58.647+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:58.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:58.647+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:58.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:58.647+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:58.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:58.648+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:59.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:58.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:59.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:58.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:59.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:58.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:52:59.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:58.903+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:52:59.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:59.295+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:59.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:59.295+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:59.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:59.295+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:52:59.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:52:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:52:59.296+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:52:59.934 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:59.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:59.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:59.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:59.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:59.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:52:59.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:52:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:52:59.618+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:59.932+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:59.932+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:59.932+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:52:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:52:59.933+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:00.631 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:00.283+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:00.631 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:00.283+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:00.632 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:00.283+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:00.632 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:00.284+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:00.913 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:53:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:53:00.824+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 4231 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:53:00.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:00.629+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:00.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:00.629+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:00.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:00.629+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:00.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:00.630+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:01.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:00.911+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:01.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:00.911+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:01.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:00.911+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:01.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:00.912+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:01.645 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:01.250+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:01.645 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:01.250+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:01.645 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:01.250+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:01.646 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:01.251+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:01.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:01.643+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:01.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:01.643+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:01.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:01.643+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:01.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:01.644+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:02.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:01.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:02.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:01.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:02.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:01.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:02.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:01.904+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:02.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:02.202+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:02.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:02.202+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:02.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:02.202+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:02.203+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:02.867 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:02.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:02.867 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:02.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:02.867 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:02.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:02.867 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:02.620+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:03.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:02.865+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:03.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:02.865+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:03.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:02.865+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:03.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:02.865+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:03.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:03.203+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:03.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:03.203+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:03.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:03.203+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:03.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:03.204+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:03.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:03.637+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:03.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:03.637+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:03.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:03.637+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:03.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:03.638+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:04.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:03.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:04.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:03.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:04.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:03.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:04.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:03.888+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:04.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:04.156+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:04.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:04.156+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:04.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:04.156+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:04.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:04.157+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:04.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:04.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:04.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:04.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:04.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:04.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:04.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:04.650+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:04.924+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:04.924+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:04.924+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:04.925+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:05.140+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:05.140+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:05.140+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:05.141+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:05.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:05.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:05.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:05.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:05.603+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:06.116 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:53:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:53:05.825+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 4324 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:53:06.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:05.910+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:06.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:05.910+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:06.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:05.910+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:06.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:05.911+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:06.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:06.114+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:06.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:06.114+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:06.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:06.114+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:06.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:06.115+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:06.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:06.591+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:06.911 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:06.591+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:06.911 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:06.591+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:06.911 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:06.592+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:06.908+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:06.909+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:06.909+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:06.909+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:07.148+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:07.148+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:07.148+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:07.149+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:07.899 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:07.562+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:07.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:07.562+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:07.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:07.562+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:07.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:07.563+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:08.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:07.897+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:08.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:07.897+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:08.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:07.897+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:07.898+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:08.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:08.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:08.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:08.136+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:08.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:08.590+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:08.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:08.590+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:08.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:08.590+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:08.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:08.591+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:09.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:08.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:09.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:08.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:09.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:08.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:09.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:08.883+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:09.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:09.143+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:09.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:09.143+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:09.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:09.143+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:09.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:09.143+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:09.853 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:09.544+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:09.853 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:09.544+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:09.853 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:09.544+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:09.853 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:09.545+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:10.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:09.851+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:10.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:09.851+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:10.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:09.851+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:10.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:09.852+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:10.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:10.176+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:10.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:10.176+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:10.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:10.176+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:10.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:10.177+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:10.829 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:10.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:10.829 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:10.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:10.829 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:10.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:10.829 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:10.532+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:11.189 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:53:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:53:10.826+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 4421 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:53:11.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:10.853+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:11.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:10.853+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:11.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:10.853+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:11.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:10.854+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:11.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:11.187+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:11.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:11.187+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:11.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:11.187+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:11.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:11.188+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:11.824 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:11.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:11.824 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:11.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:11.824 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:11.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:11.824 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:11.511+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:12.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:11.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:12.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:11.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:12.195 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:11.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:12.195 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:11.822+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:12.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:12.192+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:12.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:12.192+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:12.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:12.192+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:12.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:12.193+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:12.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:12.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:12.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:12.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:12.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:12.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:12.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:12.474+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:12.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:12.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:12.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:12.807+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:13.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:13.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:13.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:13.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:13.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:13.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:13.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:13.441+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:13.540 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:13.235+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:13.540 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:13.235+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:13.540 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:13.235+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:13.540 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:13.236+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:14.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:13.819+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:14.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:13.819+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:14.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:13.819+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:14.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:13.820+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:14.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:14.405+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:14.405+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:14.405+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:14.405+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:14.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:14.199+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:14.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:14.199+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:14.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:14.199+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:14.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:14.199+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:14.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:14.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:14.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:14.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:14.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:14.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:14.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:14.845+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:15.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:15.359+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:15.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:15.359+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:15.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:15.359+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:15.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:15.359+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:15.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:15.150+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:15.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:15.150+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:15.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:15.150+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:15.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:15.151+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:16.175 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:53:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:53:15.827+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 4522 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:53:16.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:15.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:16.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:15.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:16.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:15.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:16.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:15.808+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:16.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:16.173+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:16.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:16.173+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:16.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:16.173+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:16.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:16.174+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:16.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:16.331+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:16.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:16.331+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:16.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:16.331+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:16.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:16.332+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:17.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:16.816+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:17.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:16.816+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:17.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:16.816+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:17.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:16.817+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:17.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:17.332+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:17.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:17.332+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:17.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:17.332+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:17.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:17.333+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:17.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:17.210+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:17.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:17.210+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:17.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:17.210+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:17.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:17.211+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:17.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:17.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:17.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:17.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:17.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:17.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:17.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:17.848+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:18.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:18.303+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:18.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:18.303+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:18.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:18.303+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:18.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:18.304+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:18.219+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:18.219+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:18.219+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:18.219+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:19.189 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:18.896+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:19.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:18.896+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:19.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:18.896+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:19.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:18.897+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:19.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:19.187+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:19.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:19.187+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:19.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:19.187+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:19.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:19.188+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:19.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:19.336+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:19.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:19.336+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:19.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:19.336+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:19.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:19.336+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:20.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:19.860+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:20.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:19.860+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:20.186 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:19.860+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:20.186 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:19.861+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:20.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:20.326+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:20.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:20.326+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:20.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:20.326+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:20.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:20.326+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:20.183+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:20.183+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:20.183+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:20.184+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:21.202 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:53:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:53:20.828+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 4621 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:53:21.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:20.812+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:21.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:20.812+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:21.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:20.812+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:21.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:20.813+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:21.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:21.352+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:21.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:21.352+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:21.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:21.352+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:21.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:21.352+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:21.200+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:21.200+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:21.200+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:21.201+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:22.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:21.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:22.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:21.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:22.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:21.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:22.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:21.800+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:22.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:22.311+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:22.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:22.311+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:22.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:22.311+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:22.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:22.312+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:22.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:22.210+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:22.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:22.210+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:22.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:22.210+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:22.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:22.210+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:23.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:22.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:23.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:22.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:23.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:22.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:23.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:22.765+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:23.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:23.268+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:23.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:23.268+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:23.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:23.268+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:23.269+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:23.187+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:23.187+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:23.187+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:23.188+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:24.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:23.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:24.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:23.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:24.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:23.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:24.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:23.810+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:24.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:24.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:24.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:24.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:24.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:24.181+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:24.273+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:24.273+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:24.273+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:24.274+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:25.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:24.847+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:25.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:24.847+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:25.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:24.847+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:25.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:24.848+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:25.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:25.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:25.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:25.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:25.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:25.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:25.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:25.171+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:25.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:25.239+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:25.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:25.239+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:25.239+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:25.240+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:26.176 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:53:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:53:25.829+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 4718 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:53:26.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:25.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:26.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:25.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:26.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:25.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:26.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:25.811+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:26.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:26.207+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:26.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:26.207+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:26.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:26.207+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:26.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:26.208+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:26.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:26.174+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:26.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:26.174+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:26.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:26.174+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:26.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:26.175+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:27.189 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:26.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:27.189 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:26.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:27.189 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:26.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:27.189 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:26.765+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:27.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:27.187+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:27.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:27.187+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:27.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:27.187+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:27.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:27.188+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:27.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:27.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:27.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:27.188+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:28.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:27.738+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:28.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:27.738+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:28.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:27.738+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:28.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:27.738+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:28.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:28.181+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:28.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:28.181+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:28.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:28.181+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:28.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:28.182+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:28.176+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:28.176+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:28.176+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:28.176+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:29.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:28.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:29.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:28.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:29.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:28.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:29.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:28.719+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:29.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:29.157+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:29.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:29.157+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:29.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:29.157+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:29.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:29.158+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:29.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:29.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:29.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:29.204+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:30.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:29.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:30.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:29.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:30.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:29.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:30.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:29.737+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:30.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:30.199+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:30.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:30.199+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:30.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:30.199+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:30.200+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:30.164+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:30.164+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:30.164+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:30.165+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:31.178 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:53:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:53:30.830+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 4815 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:53:31.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:30.757+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:31.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:30.757+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:31.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:30.757+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:31.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:30.758+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:31.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:31.189+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:31.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:31.189+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:31.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:31.189+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:31.189+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:31.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:31.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:31.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:31.175+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:32.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:31.794+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:32.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:31.794+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:32.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:31.794+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:32.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:31.795+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:32.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:32.153+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:32.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:32.153+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:32.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:32.153+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:32.153+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:32.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:32.210+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:32.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:32.210+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:32.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:32.210+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:32.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:32.211+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:33.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:32.805+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:33.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:32.805+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:33.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:32.805+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:33.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:32.806+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:33.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:33.233+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:33.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:33.233+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:33.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:33.233+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:33.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:33.234+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:33.121+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:33.121+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:33.121+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:33.122+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:33.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:33.787+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:33.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:33.787+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:33.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:33.787+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:33.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:33.788+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:34.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:34.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:34.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:34.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:34.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:34.163+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:34.267+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:34.267+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:34.267+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:34.267+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:35.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:34.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:35.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:34.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:35.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:34.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:35.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:34.791+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:35.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:35.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:35.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:35.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:35.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:35.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:35.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:35.204+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:35.269+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:35.269+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:35.269+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:35.270+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:36.144 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:53:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:53:35.831+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 4922 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:53:36.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:35.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:36.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:35.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:36.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:35.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:36.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:35.791+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:36.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:36.241+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:36.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:36.241+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:36.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:36.241+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:36.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:36.241+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:36.170+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:36.170+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:36.170+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:36.171+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:37.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:36.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:37.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:36.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:37.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:36.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:37.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:36.823+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:37.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:37.223+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:37.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:37.223+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:37.223+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:37.223+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:37.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:37.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:37.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:37.202+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:38.198 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:37.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:38.198 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:37.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:38.198 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:37.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:38.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:37.808+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:38.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:38.195+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:38.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:38.195+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:38.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:38.195+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:38.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:38.196+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:38.253+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:38.253+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:38.253+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:38.254+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:39.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:38.801+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:39.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:38.801+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:39.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:38.801+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:39.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:38.802+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:39.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:39.296+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:39.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:39.296+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:39.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:39.296+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:39.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:39.296+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:39.157+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:39.157+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:39.157+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:39.158+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:40.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:39.787+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:40.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:39.787+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:40.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:39.787+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:40.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:39.787+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:40.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:40.307+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:40.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:40.307+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:40.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:40.307+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:40.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:40.308+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:40.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:40.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:40.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:40.163+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:41.175 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:53:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:53:40.830+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 5017 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:53:41.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:40.776+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:41.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:40.776+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:41.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:40.776+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:41.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:40.777+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:41.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:41.173+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:41.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:41.173+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:41.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:41.173+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:41.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:41.174+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:41.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:41.357+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:41.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:41.357+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:41.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:41.357+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:41.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:41.358+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:42.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:41.776+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:42.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:41.776+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:42.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:41.776+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:42.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:41.776+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:42.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:42.156+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:42.156+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:42.156+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:42.157+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:42.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:42.310+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:42.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:42.310+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:42.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:42.310+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:42.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:42.311+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:43.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:42.742+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:43.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:42.742+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:43.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:42.742+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:43.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:42.743+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:43.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:43.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:43.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:43.200+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:43.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:43.272+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:43.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:43.272+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:43.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:43.272+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:43.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:43.272+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:43.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:43.776+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:43.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:43.776+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:43.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:43.776+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:43.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:43.777+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:44.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:44.288+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:44.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:44.288+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:44.288+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:44.289+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:44.155+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:44.155+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:44.155+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:44.156+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:45.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:44.821+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:45.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:44.821+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:45.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:44.821+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:45.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:44.822+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:45.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:45.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:45.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:45.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:45.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:45.144+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:45.253+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:45.253+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:45.253+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:45.254+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:46.103 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:53:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:53:45.832+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 5110 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:53:46.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:45.808+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:46.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:45.808+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:46.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:45.808+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:46.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:45.809+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:46.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:46.101+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:46.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:46.101+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:46.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:46.101+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:46.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:46.102+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:46.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:46.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:46.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:46.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:46.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:46.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:46.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:46.229+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:47.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:46.800+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:47.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:46.800+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:47.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:46.800+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:47.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:46.801+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:47.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:47.093+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:47.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:47.093+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:47.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:47.093+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:47.094+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:47.215+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:47.215+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:47.215+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:47.216+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:48.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:48.046+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:48.046+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:48.046+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:48.047+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:47.836+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:47.836+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:47.836+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:47.837+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:48.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:48.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:48.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:48.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:48.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:48.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:48.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:48.230+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:49.050+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:49.050+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:49.050+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:49.050+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:48.853+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:48.853+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:48.853+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:48.854+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:49.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:49.221+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:49.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:49.221+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:49.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:49.221+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:49.221+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:49.859+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:49.859+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:49.859+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:49.860+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:50.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:50.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:50.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:50.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:50.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:50.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:50.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:50.060+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:50.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:50.222+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:50.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:50.222+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:50.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:50.222+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:50.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:50.223+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:51.082 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:53:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:53:50.833+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 5210 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:53:51.082 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:50.821+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:51.082 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:50.821+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:51.082 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:50.821+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:51.082 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:50.822+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:51.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:51.080+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:51.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:51.080+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:51.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:51.080+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:51.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:51.080+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:51.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:51.198+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:51.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:51.198+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:51.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:51.198+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:51.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:51.199+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:52.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:51.787+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:52.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:51.787+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:52.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:51.787+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:52.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:51.788+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:52.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:52.082+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:52.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:52.082+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:52.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:52.082+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:52.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:52.083+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:52.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:52.219+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:52.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:52.219+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:52.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:52.219+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:52.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:52.220+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:53.069 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:52.793+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:53.069 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:52.793+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:53.069 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:52.793+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:53.069 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:52.794+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:53.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:53.067+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:53.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:53.067+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:53.067+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:53.068+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:53.196+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:53.196+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:53.196+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:53.196+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:54.091 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:53.833+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:54.091 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:53.833+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:54.091 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:53.833+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:54.091 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:53.834+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:54.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:54.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:54.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:54.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:54.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:54.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:54.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:54.089+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:54.231+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:54.231+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:54.231+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:54.231+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:55.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:54.864+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:55.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:54.864+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:55.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:54.864+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:55.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:54.865+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:55.095+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:55.095+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:55.095+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:55.095+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:55.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:55.255+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:55.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:55.255+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:55.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:55.255+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:55.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:55.256+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:56.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:53:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:53:55.834+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 5321 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:53:56.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:56.075+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:56.075+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:56.075+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:56.075+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:55.894+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:55.894+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:55.894+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:55.895+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:56.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:56.217+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:56.217+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:56.217+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:56.218+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:57.124 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:56.861+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:57.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:56.861+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:57.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:56.861+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:57.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:56.861+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:57.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:57.122+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:57.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:57.122+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:57.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:57.122+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:57.123+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:57.213+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:57.213+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:57.213+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:57.214+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:57.871+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:57.871+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:57.871+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:57.871+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:58.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:58.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:58.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:58.106+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:58.667 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:58.244+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:58.667 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:58.244+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:58.667 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:58.244+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:58.667 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:58.245+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:58.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:58.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:58.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:58.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:58.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:53:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:58.841+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:53:59.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:59.148+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:59.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:59.148+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:59.148+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:53:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:53:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:53:59.148+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:53:59.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:59.236+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:59.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:59.236+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:59.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:59.236+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:53:59.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:53:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:53:59.236+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:00.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:59.856+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:00.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:59.856+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:00.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:59.856+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:00.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:53:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:53:59.857+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:00.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:00.216+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:00.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:00.216+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:00.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:00.216+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:00.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:00.216+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:00.124+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:00.124+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:00.124+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:00.125+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:01.100 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:54:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:54:00.835+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 5431 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:54:01.101 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:00.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:01.101 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:00.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:01.101 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:00.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:01.101 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:00.809+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:01.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:01.098+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:01.363 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:01.098+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:01.363 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:01.098+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:01.363 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:01.099+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:01.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:01.206+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:01.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:01.206+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:01.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:01.206+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:01.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:01.207+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:02.108 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:01.778+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:02.108 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:01.778+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:02.108 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:01.778+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:02.108 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:01.779+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:02.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:02.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:02.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:02.107+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:02.253+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:02.253+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:02.253+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:02.254+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:03.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:02.731+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:03.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:02.731+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:03.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:02.731+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:03.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:02.732+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:03.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:03.100+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:03.100+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:03.100+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:03.100+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:03.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:03.292+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:03.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:03.292+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:03.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:03.292+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:03.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:03.293+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:03.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:03.739+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:03.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:03.739+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:03.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:03.739+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:03.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:03.740+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:04.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:04.083+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:04.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:04.083+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:04.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:04.083+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:04.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:04.084+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:04.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:04.323+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:04.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:04.323+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:04.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:04.323+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:04.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:04.323+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:05.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:04.716+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:05.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:04.716+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:05.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:04.716+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:05.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:04.716+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:05.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:05.362+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:05.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:05.362+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:05.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:05.362+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:05.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:05.362+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:05.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:05.126+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:05.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:05.126+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:05.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:05.126+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:05.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:05.127+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:05.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:54:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:54:05.836+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 5525 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:54:05.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:05.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:05.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:05.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:05.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:05.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:05.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:05.673+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:06.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:06.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:06.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:06.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:06.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:06.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:06.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:06.140+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:06.664 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:06.358+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:06.664 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:06.358+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:06.664 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:06.358+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:06.664 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:06.359+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:06.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:06.662+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:06.662+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:06.662+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:06.663+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:07.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:07.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:07.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:07.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:07.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:07.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:07.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:07.155+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:07.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:07.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:07.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:07.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:07.385+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:07.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:07.662+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:07.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:07.662+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:07.662+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:07.662+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:08.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:08.205+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:08.205+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:08.205+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:08.206+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:08.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:08.427+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:08.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:08.427+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:08.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:08.427+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:08.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:08.427+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:08.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:08.669+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:08.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:08.669+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:08.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:08.669+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:08.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:08.670+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:09.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:09.226+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:09.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:09.226+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:09.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:09.226+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:09.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:09.227+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:09.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:09.449+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:09.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:09.449+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:09.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:09.449+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:09.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:09.450+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:09.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:09.711+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:09.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:09.711+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:09.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:09.711+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:09.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:09.711+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:10.462 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:10.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:10.462 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:10.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:10.462 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:10.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:10.462 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:10.179+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:10.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:10.460+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:10.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:10.460+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:10.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:10.460+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:10.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:10.461+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:11.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:54:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:54:10.837+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 5622 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:54:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:10.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:10.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:10.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:10.749+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:11.496 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:11.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:11.496 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:11.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:11.496 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:11.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:11.496 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:11.215+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:11.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:11.494+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:11.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:11.494+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:11.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:11.494+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:11.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:11.495+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:11.767+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:11.767+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:11.767+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:11.767+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:12.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:12.462+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:12.462+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:12.462+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:12.462+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:12.217+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:12.217+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:12.217+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:12.218+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:12.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:12.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:12.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:12.797+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:13.543 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:13.227+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:13.543 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:13.227+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:13.543 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:13.227+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:13.543 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:13.228+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:13.544 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:13.414+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:13.544 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:13.414+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:13.544 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:13.414+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:13.544 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:13.415+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:13.810+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:13.810+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:13.810+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:13.811+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:14.257+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:14.257+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:14.257+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:14.258+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:14.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:14.434+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:14.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:14.434+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:14.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:14.434+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:14.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:14.435+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:15.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:14.810+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:14.810+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:14.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:14.811+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:15.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:15.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:15.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:15.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:15.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:15.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:15.308+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:15.463+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:15.463+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:15.463+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:15.463+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:16.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:15.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:16.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:15.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:16.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:15.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:16.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:15.812+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:16.080 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:54:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:54:15.838+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 5715 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:54:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:16.350+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:16.350+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:16.350+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:16.351+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:16.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:16.466+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:16.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:16.466+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:16.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:16.466+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:16.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:16.467+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:17.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:16.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:17.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:16.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:17.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:16.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:17.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:16.816+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:17.485 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:17.346+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:17.485 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:17.346+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:17.485 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:17.346+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:17.485 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:17.347+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:17.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:17.482+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:17.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:17.482+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:17.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:17.482+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:17.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:17.483+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:17.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:17.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:17.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:17.858+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:18.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:18.297+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:18.297+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:18.297+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:18.298+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:18.525+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:18.525+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:18.525+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:18.526+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:19.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:18.892+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:19.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:18.892+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:19.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:18.892+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:19.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:18.892+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:19.576 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:19.282+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:19.576 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:19.282+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:19.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:19.282+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:19.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:19.283+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:19.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:19.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:19.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:19.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:19.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:19.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:19.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:19.575+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:20.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:19.937+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:20.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:19.937+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:20.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:19.937+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:20.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:19.938+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:20.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:20.314+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:20.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:20.314+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:20.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:20.314+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:20.594 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:20.315+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:20.842 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:20.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:20.842 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:20.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:20.842 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:20.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:20.842 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:20.575+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:21.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:54:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:54:20.839+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 5811 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:54:21.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:20.899+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:20.899+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:20.899+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:20.900+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:21.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:21.312+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:21.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:21.312+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:21.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:21.312+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:21.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:21.312+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:21.866 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:21.538+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:21.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:21.538+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:21.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:21.538+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:21.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:21.538+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:22.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:21.863+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:22.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:21.863+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:22.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:21.863+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:21.864+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:22.532 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:22.268+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:22.532 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:22.268+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:22.532 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:22.268+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:22.532 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:22.268+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:22.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:22.529+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:22.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:22.529+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:22.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:22.529+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:22.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:22.529+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:23.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:22.899+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:23.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:22.899+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:23.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:22.899+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:23.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:22.899+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:23.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:23.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:23.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:23.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:23.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:23.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:23.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:23.221+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:23.912 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:23.537+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:23.912 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:23.537+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:23.912 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:23.537+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:23.913 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:23.537+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:24.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:23.909+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:24.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:23.909+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:24.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:23.909+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:24.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:23.910+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:24.225+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:24.225+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:24.225+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:24.226+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:24.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:24.548+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:24.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:24.548+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:24.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:24.548+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:24.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:24.549+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:24.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:24.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:24.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:25.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:24.880+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:25.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:25.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:25.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:25.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:25.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:25.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:25.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:25.251+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:25.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:25.536+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:25.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:25.536+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:25.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:25.536+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:25.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:25.536+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:26.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:54:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:54:25.839+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 5904 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:54:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:25.856+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:25.856+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:25.856+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:25.857+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:26.509 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:26.215+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:26.509 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:26.215+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:26.509 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:26.215+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:26.509 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:26.216+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:26.856 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:26.507+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:26.856 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:26.507+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:26.856 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:26.507+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:26.856 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:26.508+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:27.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:26.854+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:27.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:26.854+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:27.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:26.854+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:27.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:26.854+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:27.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:27.169+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:27.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:27.169+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:27.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:27.169+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:27.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:27.170+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:27.816 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:27.491+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:27.816 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:27.491+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:27.816 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:27.491+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:27.816 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:27.491+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:28.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:27.814+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:28.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:27.814+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:28.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:27.814+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:28.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:27.815+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:28.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:28.169+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:28.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:28.169+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:28.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:28.169+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:28.449 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:28.170+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:28.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:28.446+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:28.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:28.446+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:28.446+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:28.447+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:29.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:28.793+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:29.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:28.793+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:29.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:28.793+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:29.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:28.793+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:29.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:29.126+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:29.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:29.126+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:29.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:29.126+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:29.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:29.127+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:29.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:29.407+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:29.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:29.407+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:29.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:29.407+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:29.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:29.408+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:30.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:29.773+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:30.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:29.773+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:30.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:29.773+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:30.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:29.773+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:30.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:30.117+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:30.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:30.117+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:30.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:30.117+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:30.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:30.117+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:30.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:30.441+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:30.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:30.441+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:30.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:30.441+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:30.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:30.442+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:31.134 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:54:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:54:30.841+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 6001 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:54:31.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:30.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:31.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:30.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:31.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:30.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:31.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:30.816+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:31.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:31.132+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:31.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:31.132+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:31.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:31.132+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:31.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:31.133+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:31.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:31.450+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:31.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:31.450+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:31.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:31.450+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:31.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:31.451+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:32.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:31.801+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:32.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:31.801+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:32.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:31.801+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:32.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:31.801+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:32.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:32.110+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:32.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:32.110+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:32.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:32.110+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:32.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:32.110+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:32.821 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:32.494+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:32.821 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:32.494+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:32.821 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:32.494+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:32.821 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:32.494+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:33.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:32.819+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:33.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:32.819+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:33.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:32.819+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:33.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:32.819+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:33.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:33.137+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:33.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:33.137+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:33.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:33.137+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:33.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:33.138+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:33.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:33.487+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:33.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:33.487+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:33.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:33.487+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:33.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:33.488+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:34.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:33.832+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:34.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:33.832+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:34.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:33.832+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:34.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:33.832+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:34.457 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:34.129+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:34.457 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:34.129+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:34.457 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:34.129+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:34.457 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:34.129+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:34.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:34.455+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:34.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:34.455+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:34.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:34.455+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:34.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:34.456+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:35.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:34.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:35.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:34.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:35.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:34.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:35.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:34.793+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:35.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:35.157+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:35.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:35.157+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:35.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:35.157+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:35.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:35.158+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:35.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:35.483+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:35.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:35.483+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:35.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:35.483+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:35.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:35.484+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:36.119 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:54:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:54:35.842+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 6093 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:54:36.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:35.755+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:36.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:35.755+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:36.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:35.755+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:36.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:35.756+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:36.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:36.117+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:36.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:36.117+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:36.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:36.117+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:36.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:36.117+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:36.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:36.454+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:36.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:36.454+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:36.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:36.454+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:36.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:36.454+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:37.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:36.718+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:37.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:36.718+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:37.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:36.718+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:37.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:36.718+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:37.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:37.101+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:37.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:37.101+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:37.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:37.101+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:37.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:37.102+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:37.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:37.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:37.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:37.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:37.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:37.749+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:37.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:37.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:37.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:37.503+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:38.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:38.078+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:38.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:38.078+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:38.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:38.078+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:38.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:38.078+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:38.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:38.462+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:38.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:38.462+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:38.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:38.462+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:38.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:38.463+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:39.117 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:38.789+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:39.117 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:38.789+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:39.117 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:38.789+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:39.117 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:38.789+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:39.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:39.115+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:39.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:39.115+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:39.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:39.115+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:39.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:39.116+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:39.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:39.507+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:39.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:39.507+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:39.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:39.507+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:39.508+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:40.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:39.742+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:40.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:39.742+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:40.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:39.742+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:40.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:39.743+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:40.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:40.123+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:40.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:40.123+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:40.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:40.123+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:40.124+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:40.846 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:40.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:40.846 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:40.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:40.846 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:40.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:40.846 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:40.783+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:40.846 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:40.538+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:40.846 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:40.538+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:40.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:40.538+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:40.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:40.539+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:41.152 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:54:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:54:40.843+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 6188 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:54:41.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:41.150+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:41.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:41.150+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:41.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:41.150+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:41.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:41.151+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:41.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:41.735+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:41.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:41.735+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:41.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:41.735+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:41.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:41.736+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:41.547+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:41.547+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:41.547+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:41.548+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:42.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:42.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:42.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:42.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:42.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:42.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:42.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:42.144+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:42.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:42.687+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:42.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:42.687+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:42.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:42.687+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:42.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:42.688+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:42.504+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:42.504+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:42.504+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:42.504+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:43.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:43.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:43.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:43.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:43.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:43.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:43.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:43.194+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:43.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:43.690+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:43.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:43.690+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:43.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:43.690+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:43.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:43.690+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:43.536+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:43.536+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:43.536+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:43.537+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:44.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:44.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:44.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:44.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:44.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:44.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:44.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:44.184+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:44.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:44.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:44.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:44.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:44.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:44.669+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:44.522+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:44.522+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:44.522+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:44.522+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:45.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:45.200+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:45.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:45.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:45.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:45.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:45.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:45.201+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:45.847 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:45.651+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:45.847 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:45.651+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:45.847 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:45.651+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:45.847 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:45.652+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:45.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:45.497+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:45.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:45.497+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:45.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:45.497+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:45.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:45.498+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:54:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:54:45.844+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 6281 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:54:46.533 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:46.250+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:46.533 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:46.250+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:46.533 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:46.250+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:46.533 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:46.251+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:46.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:46.694+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:46.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:46.694+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:46.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:46.694+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:46.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:46.695+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:46.531+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:46.531+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:46.531+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:46.532+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:47.502 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:47.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:47.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:47.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:47.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:47.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:47.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:47.215+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:47.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:47.743+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:47.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:47.743+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:47.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:47.743+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:47.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:47.744+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:47.500+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:47.500+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:47.500+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:47.501+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:48.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:48.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:48.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:48.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:48.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:48.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:48.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:48.189+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:48.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:48.539+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:48.539+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:48.539+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:48.540+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:48.700+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:48.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:48.700+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:48.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:48.700+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:48.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:48.701+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:49.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:49.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:49.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:49.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:49.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:49.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:49.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:49.178+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:49.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:49.660+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:49.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:49.660+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:49.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:49.660+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:49.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:49.660+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:49.523+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:49.523+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:49.523+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:49.524+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:50.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:50.197+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:50.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:50.197+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:50.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:50.197+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:50.198+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:50.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:50.708+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:50.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:50.708+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:50.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:50.708+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:50.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:50.709+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:50.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:50.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:50.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:50.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:50.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:50.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:50.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:50.479+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:51.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:54:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:54:50.845+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 6378 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:54:51.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:51.223+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:51.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:51.223+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:51.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:51.223+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:51.224+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:51.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:51.467+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:51.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:51.467+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:51.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:51.467+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:51.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:51.468+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:52.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:51.758+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:51.758+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:51.758+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:51.759+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:52.515 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:52.216+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:52.515 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:52.216+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:52.515 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:52.216+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:52.515 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:52.217+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:52.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:52.709+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:52.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:52.709+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:52.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:52.709+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:52.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:52.710+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:52.513+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:52.513+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:52.513+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:52.514+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:53.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:53.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:53.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:53.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:53.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:53.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:53.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:53.176+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:53.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:53.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:53.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:53.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:53.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:53.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:53.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:53.728+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:53.532+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:53.532+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:53.532+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:53.533+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:54.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:54.200+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:54.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:54.200+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:54.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:54.200+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:54.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:54.201+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:54.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:54.708+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:54.708+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:54.708+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:54.709+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:54.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:54.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:54.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:54.502+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:55.531 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:55.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:55.532 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:55.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:55.532 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:55.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:55.532 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:55.237+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:55.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:55.747+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:55.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:55.747+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:55.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:55.747+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:55.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:55.748+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:55.849 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:55.529+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:55.849 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:55.529+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:55.849 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:55.529+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:55.850 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:55.530+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:56.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:54:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:54:55.846+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 6470 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:54:56.580 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:56.263+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:56.580 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:56.263+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:56.580 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:56.263+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:56.580 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:56.264+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:56.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:56.729+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:56.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:56.729+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:56.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:56.729+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:56.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:56.730+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:56.578+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:56.578+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:56.578+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:56.579+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:57.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:57.294+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:57.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:57.294+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:57.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:57.294+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:57.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:57.295+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:57.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:57.713+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:57.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:57.713+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:57.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:57.713+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:57.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:57.714+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:57.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:57.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:57.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:57.576+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:58.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:58.296+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:58.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:58.296+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:58.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:58.296+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:58.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:58.297+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:58.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:58.591+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:58.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:58.591+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:58.591+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:58.591+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:58.754+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:58.754+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:58.754+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:58.755+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:54:59.547 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:59.282+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:59.547 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:59.282+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:59.547 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:59.282+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:54:59.548 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:54:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:54:59.283+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:59.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:59.545+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:59.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:59.545+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:59.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:59.545+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:54:59.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:54:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:54:59.546+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:54:59.824 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:59.728+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:59.824 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:59.728+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:59.824 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:59.728+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:54:59.824 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:54:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:54:59.729+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:00.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:00.262+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:00.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:00.262+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:00.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:00.262+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:00.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:00.263+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:00.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:00.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:00.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:00.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:00.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:00.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:00.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:00.503+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:00.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:55:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:55:00.847+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 6566 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:55:00.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:00.711+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:00.711+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:00.711+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:00.712+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:01.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:01.241+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:01.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:01.241+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:01.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:01.241+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:01.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:01.242+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:01.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:01.708+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:01.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:01.708+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:01.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:01.708+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:01.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:01.709+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:01.520+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:01.520+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:01.520+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:01.521+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:02.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:02.227+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:02.529 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:02.227+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:02.529 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:02.227+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:02.529 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:02.228+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:02.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:02.526+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:02.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:02.526+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:02.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:02.526+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:02.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:02.527+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:02.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:02.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:02.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:02.754+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:03.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:03.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:03.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:03.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:03.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:03.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:03.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:03.179+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:03.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:03.513+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:03.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:03.513+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:03.513+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:03.513+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:03.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:03.760+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:03.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:03.760+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:03.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:03.760+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:03.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:03.761+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:04.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:04.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:04.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:04.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:04.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:04.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:04.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:04.159+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:04.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:04.464+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:04.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:04.464+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:04.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:04.464+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:04.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:04.465+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:05.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:04.777+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:05.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:04.777+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:05.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:04.777+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:05.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:04.778+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:05.445 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:05.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:05.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:05.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:05.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:05.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:05.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:05.175+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:05.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:05.443+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:05.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:05.443+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:05.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:05.443+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:05.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:05.444+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:06.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:05.803+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:06.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:05.803+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:06.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:05.803+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:06.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:05.804+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:06.206 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:55:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:55:05.848+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 6660 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:55:06.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:06.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:06.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:06.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:06.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:06.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:06.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:06.204+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:06.785 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:06.473+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:06.785 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:06.473+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:06.785 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:06.473+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:06.785 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:06.474+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:07.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:06.783+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:07.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:06.783+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:07.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:06.783+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:07.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:06.784+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:07.462 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:07.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:07.462 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:07.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:07.462 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:07.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:07.462 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:07.155+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:07.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:07.459+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:07.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:07.459+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:07.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:07.459+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:07.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:07.460+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:08.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:07.806+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:08.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:07.806+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:08.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:07.806+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:08.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:07.807+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:08.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:08.128+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:08.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:08.128+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:08.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:08.128+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:08.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:08.129+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:08.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:08.443+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:08.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:08.443+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:08.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:08.443+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:08.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:08.444+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:09.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:08.843+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:09.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:08.843+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:09.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:08.843+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:09.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:08.844+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:09.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:09.155+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:09.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:09.155+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:09.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:09.155+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:09.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:09.156+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:09.863 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:09.487+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:09.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:09.487+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:09.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:09.488+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:09.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:09.488+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:10.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:09.860+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:10.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:09.860+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:10.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:09.860+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:10.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:09.861+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:10.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:10.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:10.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:10.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:10.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:10.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:10.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:10.172+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:10.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:10.465+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:10.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:10.465+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:10.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:10.465+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:10.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:10.466+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:11.160 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:55:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:55:10.849+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 6757 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:55:11.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:10.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:11.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:10.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:11.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:10.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:11.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:10.876+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:11.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:11.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:11.449 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:11.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:11.449 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:11.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:11.449 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:11.158+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:11.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:11.445+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:11.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:11.445+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:11.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:11.445+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:11.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:11.446+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:12.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:11.912+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:44.743231+0000 front 2024-09-20T19:44:44.743263+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:12.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:11.912+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:44.743294+0000 front 2024-09-20T19:44:44.743359+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:12.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:11.912+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:44.743334+0000 front 2024-09-20T19:44:44.743318+0000 (oldest deadline 2024-09-20T19:45:08.843236+0000) 2024-09-20T19:55:12.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:11.913+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:12.460 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:12.204+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:12.461 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:12.204+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:12.461 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:12.204+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:12.461 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:12.205+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:12.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:12.458+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:41.579169+0000 front 2024-09-20T19:44:41.579198+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:12.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:12.458+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:41.579219+0000 front 2024-09-20T19:44:41.579189+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:12.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:12.458+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:41.579211+0000 front 2024-09-20T19:44:41.579158+0000 (oldest deadline 2024-09-20T19:45:06.879134+0000) 2024-09-20T19:55:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:12.459+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:13.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:12.907+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:13.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:13.208+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:13.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:13.208+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:13.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:13.208+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:13.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:13.208+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:13.896 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:13.472+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:14.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:13.894+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:14.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:14.437+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:14.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:14.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:14.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:14.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:14.204+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:15.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:14.885+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:15.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:15.414+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:15.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:15.230+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:15.230+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:15.230+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:15.231+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:16.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:55:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:55:15.851+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 6850 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:55:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:15.936+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:16.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:16.368+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:16.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:16.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:16.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:16.252+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:17.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:16.942+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:17.404+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:17.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:17.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:17.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:17.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:17.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:17.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:17.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:17.215+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:18.442 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:17.971+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:18.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:18.181+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:18.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:18.181+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:18.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:18.181+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:18.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:18.182+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:18.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:18.440+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:18.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T19:55:18.836+0000 7fada6d49640 0 monclient(hunting): authenticate timed out after 300 2024-09-20T19:55:18.840 INFO:teuthology.orchestra.run.smithi114.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-09-20T19:55:19.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:19.010+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:19.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:19.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:19.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:19.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:19.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:19.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:19.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:19.200+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:19.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-09-20T19:55:19.849 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T19:55:19.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:19.477+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:20.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:19.966+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:20.509 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:20.222+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:20.509 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:20.222+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:20.509 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:20.222+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:20.510 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:20.223+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:20.854 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:20.507+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:21.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:55:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:55:20.852+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 6945 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:55:21.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:20.922+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:21.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:21.495+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:21.497 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:21.235+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:21.497 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:21.235+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:21.497 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:21.235+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:21.498 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:21.235+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:22.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:21.878+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:22.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:22.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:22.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:22.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:22.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:22.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:22.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:22.189+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:22.895 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:22.538+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:23.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:22.893+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:23.494 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:23.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:23.494 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:23.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:23.494 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:23.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:23.494 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:23.215+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:23.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:23.890+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:23.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:23.492+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:24.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:24.190+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 since back 2024-09-20T19:44:45.362528+0000 front 2024-09-20T19:44:45.362362+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:24.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:24.190+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 since back 2024-09-20T19:44:45.362743+0000 front 2024-09-20T19:44:45.362607+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:24.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:24.190+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 since back 2024-09-20T19:44:45.362642+0000 front 2024-09-20T19:44:45.362395+0000 (oldest deadline 2024-09-20T19:45:06.462154+0000) 2024-09-20T19:55:24.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:24.190+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:24.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:24.499+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:24.937+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:25.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:25.157+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:25.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:25.472+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:26.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:25.959+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:26.205 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:55:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:55:25.853+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 7038 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:55:26.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:26.202+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:26.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:26.451+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:27.432 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:26.996+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:27.432 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:27.186+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:27.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:27.430+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:28.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:28.162+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:28.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:27.955+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:28.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:28.423+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:29.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:28.934+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:29.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:29.144+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:29.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:29.461+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:30.406 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:29.969+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:30.406 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:30.193+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:30.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:30.465+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:31.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:55:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:55:30.854+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 7134 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:55:31.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:30.922+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:31.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:31.454+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:31.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:31.243+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:32.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:31.922+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:32.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:32.286+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:32.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:32.476+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:33.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:32.882+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:33.548 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:33.280+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:33.549 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:33.477+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:34.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:33.855+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:34.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:34.297+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:34.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:34.520+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:34.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:34.520+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:34.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:34.520+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:34.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:34.521+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:35.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:34.896+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:35.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:35.306+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:35.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:35.499+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:35.499+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:35.499+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:35.499+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:36.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:55:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:55:35.855+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 7226 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:55:36.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:35.925+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:36.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:36.351+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:36.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:36.486+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:36.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:36.486+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:36.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:36.486+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:36.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:36.487+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:37.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:36.880+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:37.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:37.311+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:37.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:37.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:37.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:37.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:37.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:37.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:37.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:37.502+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:38.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:37.854+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:38.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:37.854+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:38.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:37.854+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:38.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:37.855+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:38.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:38.295+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:38.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:38.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:38.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:38.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:38.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:38.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:38.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:38.478+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:38.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:38.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:38.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:38.858+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:39.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:39.315+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:39.461+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:39.461+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:39.461+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:39.461+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:40.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:39.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:40.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:39.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:40.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:39.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:40.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:39.863+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:40.624 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:40.302+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:40.624 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:40.493+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:40.624 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:40.493+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:40.624 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:40.493+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:40.624 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:40.494+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:40.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:55:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:55:40.856+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 7321 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:55:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:40.853+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:40.853+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:40.853+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:40.854+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:41.532 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:41.253+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:41.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:41.530+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:41.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:41.530+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:41.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:41.530+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:41.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:41.531+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:42.209 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:41.868+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:42.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:41.868+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:42.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:41.868+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:42.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:41.868+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:42.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:42.207+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:42.837 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:42.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:42.837 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:42.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:42.837 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:42.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:42.837 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:42.572+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:43.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:42.835+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:43.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:42.835+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:43.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:42.835+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:43.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:42.835+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:43.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:43.168+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:43.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:43.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:43.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:43.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:43.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:43.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:43.863 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:43.601+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:44.150 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:43.860+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:44.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:43.860+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:44.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:43.860+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:44.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:43.861+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:44.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:44.148+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:44.884 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:44.579+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:44.884 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:44.579+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:44.884 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:44.579+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:44.884 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:44.579+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:45.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:44.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:45.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:44.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:45.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:44.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:45.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:44.882+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:45.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:45.163+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:45.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:55:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:55:45.857+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 7414 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:55:45.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:45.609+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:45.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:45.609+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:45.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:45.609+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:45.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:45.610+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:46.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:45.904+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:46.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:45.904+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:46.124 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:45.904+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:46.124 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:45.905+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:46.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:46.121+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:46.904 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:46.649+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:46.904 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:46.649+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:46.904 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:46.649+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:46.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:46.650+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:47.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:47.085+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:47.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:46.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:47.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:46.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:46.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:46.903+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:47.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:47.873+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:47.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:47.873+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:47.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:47.873+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:47.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:47.874+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:47.685+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:47.685+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:47.685+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:47.685+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:48.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:48.037+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:48.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:48.037+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:48.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:48.037+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:48.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:48.037+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:48.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:48.672+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:48.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:48.672+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:48.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:48.672+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:48.672+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:48.868+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:48.868+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:48.868+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:48.868+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:49.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:49.053+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:49.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:49.053+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:49.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:49.053+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:49.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:49.054+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:49.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:49.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:49.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:49.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:49.840+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:49.644+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:49.644+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:49.644+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:49.644+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:50.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:50.030+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:50.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:50.030+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:50.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:50.030+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:50.030+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:50.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:55:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:55:50.859+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 7510 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:55:50.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:50.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:50.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:50.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:50.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:50.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:50.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:50.796+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:50.645+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:50.645+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:50.645+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:50.646+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:51.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:51.065+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:51.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:51.065+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:51.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:51.065+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:51.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:51.065+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:51.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:51.797+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:51.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:51.797+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:51.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:51.797+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:51.798+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:51.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:51.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:51.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:51.692+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:52.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:52.037+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:52.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:52.037+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:52.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:52.037+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:52.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:52.038+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:52.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:52.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:52.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:52.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:52.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:52.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:52.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:52.792+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:52.709+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:52.709+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:52.709+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:52.709+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:53.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:53.034+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:53.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:53.034+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:53.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:53.034+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:53.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:53.035+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:53.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:53.719+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:53.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:53.719+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:53.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:53.719+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:53.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:53.720+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:53.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:53.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:53.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:53.765+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:54.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:53.986+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:54.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:53.986+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:54.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:53.986+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:54.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:53.987+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:54.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:54.755+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:54.755+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:54.755+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:54.756+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:54.687+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:54.687+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:54.687+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:54.688+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:55.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:55.025+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:55.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:55.025+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:55.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:55.025+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:55.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:55.026+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:56.027 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:55:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:55:55.859+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 7602 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:55:56.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:55.732+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:56.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:55.732+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:56.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:55.732+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:56.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:55.732+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:56.028 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:55.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:56.028 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:55.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:56.028 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:55.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:56.028 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:55.727+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:56.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:56.024+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:56.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:56.024+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:56.368 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:56.024+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:56.368 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:56.025+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:57.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:56.714+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:57.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:56.714+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:57.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:56.714+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:57.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:56.715+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:57.074 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:56.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:57.074 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:56.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:57.074 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:56.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:57.074 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:56.728+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:57.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:57.070+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:57.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:57.070+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:57.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:57.070+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:57.071+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:57.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:57.706+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:57.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:57.706+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:57.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:57.706+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:57.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:57.707+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:57.709+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:57.709+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:57.709+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:57.709+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:58.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:58.109+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:58.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:58.109+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:58.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:58.109+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:58.110+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:58.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:58.685+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:58.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:58.685+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:58.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:58.685+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:58.686+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:58.721+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:58.721+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:58.721+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:58.722+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:59.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:59.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:59.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:59.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:59.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:59.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:55:59.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:55:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:55:59.144+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:55:59.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:59.712+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:59.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:59.712+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:59.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:59.712+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:55:59.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:55:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:55:59.713+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:55:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:59.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:59.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:59.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:55:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:55:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:55:59.692+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:00.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:00.107+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:00.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:00.107+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:00.107+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:00.108+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:00.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:56:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:56:00.861+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 7698 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:56:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:00.762+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:00.762+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:00.762+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:00.763+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:00.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:00.696+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:00.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:00.696+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:00.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:00.696+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:00.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:00.697+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:01.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:01.110+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:01.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:01.110+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:01.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:01.110+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:01.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:01.111+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:01.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:01.803+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:01.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:01.803+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:01.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:01.803+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:01.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:01.804+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:01.655+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:01.655+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:01.655+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:01.656+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:02.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:02.131+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:02.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:02.131+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:02.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:02.131+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:02.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:02.132+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:02.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:02.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:02.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:02.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:02.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:02.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:02.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:02.618+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:02.758+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:02.758+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:02.758+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:02.758+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:03.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:03.096+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:03.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:03.096+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:03.096+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:03.097+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:03.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:03.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:03.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:03.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:03.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:03.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:03.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:03.764+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:03.585+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:03.585+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:03.585+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:03.586+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:04.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:04.086+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:04.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:04.086+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:04.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:04.086+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:04.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:04.086+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:04.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:04.802+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:04.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:04.802+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:04.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:04.802+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:04.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:04.803+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:04.609+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:04.609+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:04.609+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:04.610+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:05.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:05.077+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:05.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:05.077+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:05.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:05.077+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:05.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:05.078+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:05.864 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:05.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:05.865 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:05.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:05.865 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:05.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:05.865 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:05.796+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:05.865 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:05.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:05.865 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:05.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:05.865 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:05.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:05.865 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:05.577+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:06.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:56:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:56:05.862+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 7791 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:56:06.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:06.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:06.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:06.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:06.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:06.059+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:06.619 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:06.617+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:06.619 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:06.617+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:06.619 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:06.617+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:06.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:06.617+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:06.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:06.816+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:06.816+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:06.816+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:06.817+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:07.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:07.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:07.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:07.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:07.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:07.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:07.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:07.014+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:07.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:07.798+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:07.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:07.798+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:07.798+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:07.799+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:07.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:07.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:07.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:07.600+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:08.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:08.040+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:08.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:08.040+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:08.040+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:08.041+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:08.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:08.808+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:08.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:08.808+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:08.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:08.808+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:08.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:08.809+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:08.573+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:08.573+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:08.573+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:08.574+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:09.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:09.007+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:09.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:09.007+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:09.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:09.007+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:09.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:09.008+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:09.841 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:09.551+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:09.841 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:09.551+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:09.841 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:09.551+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:09.842 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:09.551+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:10.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:09.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:10.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:09.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:10.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:09.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:10.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:09.839+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:09.997+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:09.997+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:09.997+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:09.997+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:10.817 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:10.557+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:10.818 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:10.557+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:10.818 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:10.557+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:10.818 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:10.558+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:11.176 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:56:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:56:10.863+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 7887 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:56:11.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:10.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:11.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:10.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:11.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:10.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:11.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:10.816+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:11.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:10.954+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:11.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:10.954+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:11.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:10.954+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:11.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:10.955+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:11.854 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:11.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:11.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:11.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:11.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:11.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:11.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:11.572+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:12.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:11.971+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:12.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:11.971+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:11.971+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:11.972+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:11.849+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:11.849+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:11.849+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:11.849+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:12.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:12.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:12.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:12.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:12.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:12.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:12.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:12.576+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:13.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:12.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:13.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:12.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:12.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:12.926+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:12.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:12.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:12.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:12.859+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:13.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:13.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:13.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:13.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:13.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:13.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:13.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:13.577+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:13.963+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:13.963+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:13.963+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:14.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:13.964+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:13.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:13.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:13.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:13.862+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:14.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:14.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:14.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:14.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:14.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:14.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:14.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:14.822+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:14.617+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:14.617+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:14.617+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:14.617+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:15.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:14.967+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:15.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:14.967+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:15.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:14.967+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:15.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:14.968+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:15.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:56:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:56:15.865+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 7979 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:56:15.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:15.639+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:15.639+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:15.639+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:15.640+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:15.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:15.806+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:15.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:15.806+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:15.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:15.806+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:15.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:15.807+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:16.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:16.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:16.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:16.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:16.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:16.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:16.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:16.014+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:16.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:16.658+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:16.658+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:16.658+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:16.659+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:16.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:16.770+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:16.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:16.770+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:16.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:16.770+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:16.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:16.771+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:17.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:16.971+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:17.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:16.971+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:17.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:16.971+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:17.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:16.972+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:17.765 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:17.659+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:17.765 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:17.659+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:17.765 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:17.659+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:17.765 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:17.660+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:18.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:17.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:18.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:17.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:18.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:17.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:17.929+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:17.763+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:17.763+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:17.763+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:17.763+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:18.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:18.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:18.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:18.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:18.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:18.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:18.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:18.751+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:18.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:18.645+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:18.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:18.645+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:18.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:18.645+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:18.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:18.646+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:19.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:18.954+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:19.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:18.954+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:19.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:18.954+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:19.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:18.955+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:19.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:19.800+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:19.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:19.800+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:19.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:19.800+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:19.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:19.801+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:19.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:19.675+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:19.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:19.675+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:19.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:19.675+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:19.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:19.676+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:20.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:19.929+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:20.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:19.930+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:20.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:19.930+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:20.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:19.930+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:20.957 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:56:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:56:20.866+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 8073 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:56:20.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:20.786+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:20.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:20.786+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:20.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:20.786+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:20.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:20.787+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:20.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:20.626+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:20.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:20.626+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:20.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:20.626+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:20.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:20.627+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:21.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:20.955+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:21.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:20.955+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:21.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:20.955+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:21.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:20.956+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:21.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:21.676+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:21.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:21.676+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:21.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:21.676+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:21.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:21.677+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:21.758+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:21.758+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:21.758+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:21.758+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:22.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:22.005+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:22.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:22.005+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:22.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:22.005+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:22.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:22.006+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:22.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:22.676+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:22.676+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:22.676+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:22.676+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:22.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:22.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:22.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:22.781+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:23.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:23.014+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:23.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:23.014+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:23.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:23.014+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:23.015+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:23.698+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:23.698+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:23.698+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:23.698+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:23.749+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:23.749+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:23.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:23.749+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:23.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:23.750+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:24.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:24.022+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:24.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:24.022+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:24.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:24.022+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:24.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:24.022+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:25.054 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:24.770+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:25.054 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:24.770+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:25.054 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:24.770+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:25.054 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:24.771+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:25.055 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:24.719+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:25.055 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:24.719+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:25.055 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:24.719+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:25.055 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:24.720+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:25.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:25.052+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:25.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:25.052+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:25.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:25.052+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:25.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:25.053+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:26.015 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:56:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:56:25.868+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 8166 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:56:26.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:25.719+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:26.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:25.719+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:26.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:25.719+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:26.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:25.719+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:26.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:25.801+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:26.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:25.801+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:26.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:25.801+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:26.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:25.802+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:26.363 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:26.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:26.363 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:26.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:26.363 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:26.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:26.364 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:26.014+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:26.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:26.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:26.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:26.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:26.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:26.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:26.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:26.816+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:26.675+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:26.675+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:26.675+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:26.675+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:27.006+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:27.006+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:27.006+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:27.007+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:27.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:27.861+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:27.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:27.861+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:27.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:27.861+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:27.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:27.862+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:27.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:27.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:27.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:27.689+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:28.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:27.976+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:28.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:27.976+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:28.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:27.976+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:28.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:27.976+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:28.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:28.816+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:28.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:28.816+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:28.816+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:28.817+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:28.710+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:28.710+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:28.710+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:28.711+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:29.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:28.972+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:29.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:28.972+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:29.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:28.972+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:29.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:28.973+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:30.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:29.959+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:30.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:29.959+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:29.959+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:29.960+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:29.806+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:29.806+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:29.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:29.807+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:29.719+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:29.719+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:30.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:29.719+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:30.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:29.719+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:30.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:56:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:56:30.869+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 8263 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:56:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:30.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:30.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:30.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:30.834+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:30.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:30.699+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:30.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:30.699+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:30.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:30.699+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:30.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:30.700+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:31.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:31.005+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:31.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:31.005+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:31.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:31.005+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:31.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:31.006+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:32.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:31.968+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:32.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:31.968+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:32.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:31.968+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:31.968+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:31.724+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:31.724+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:31.724+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:31.725+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:31.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:31.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:31.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:31.842+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:33.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:32.793+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:33.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:32.793+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:33.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:32.793+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:33.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:32.793+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:32.940+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:32.940+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:32.940+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:32.941+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:32.743+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:32.743+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:32.743+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:32.743+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:33.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:33.757+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:33.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:33.757+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:33.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:33.757+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:33.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:33.758+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:33.911 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:33.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:33.911 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:33.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:33.912 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:33.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:33.912 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:33.773+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:34.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:33.909+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:34.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:33.909+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:34.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:33.909+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:34.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:33.910+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:35.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:34.735+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:35.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:34.735+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:35.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:34.735+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:35.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:34.736+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:34.754+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:34.754+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:34.754+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:34.755+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:34.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:34.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:34.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:34.929+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:36.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:56:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:56:35.870+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 8355 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:56:36.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:35.962+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:36.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:35.962+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:36.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:35.962+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:36.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:35.962+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:35.795+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:35.795+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:35.795+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:35.796+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:35.764+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:35.764+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:35.764+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:35.765+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:36.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:36.808+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:36.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:36.808+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:36.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:36.808+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:36.919 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:36.808+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:36.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:36.762+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:36.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:36.762+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:36.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:36.762+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:36.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:36.763+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:37.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:36.916+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:37.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:36.916+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:37.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:36.916+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:37.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:36.917+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:38.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:37.952+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:38.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:37.952+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:38.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:37.952+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:38.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:37.953+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:37.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:37.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:37.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:37.781+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:37.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:37.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:37.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:38.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:37.811+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:38.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:38.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:38.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:38.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:38.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:38.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:38.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:38.793+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:38.991 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:38.735+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:38.991 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:38.735+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:38.991 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:38.735+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:38.991 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:38.736+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:39.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:38.988+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:39.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:38.988+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:39.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:38.988+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:38.988+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:40.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:39.736+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:40.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:39.736+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:39.736+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:39.736+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:39.781+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:39.781+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:39.781+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:39.782+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:40.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:39.976+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:40.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:39.976+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:40.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:39.976+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:40.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:39.976+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:56:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:56:40.870+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 8450 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:56:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:40.784+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:40.784+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:40.784+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:40.784+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:41.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:40.970+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:41.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:40.970+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:41.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:40.970+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:41.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:40.970+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:41.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:40.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:41.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:40.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:41.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:40.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:41.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:40.752+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:42.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:41.949+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:42.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:41.949+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:42.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:41.949+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:42.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:41.949+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:41.759+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:41.759+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:41.759+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:41.760+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:42.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:41.741+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:42.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:41.741+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:42.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:41.741+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:42.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:41.742+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:43.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:42.920+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:43.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:42.920+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:43.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:42.920+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:43.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:42.921+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:42.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:42.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:42.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:42.807+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:42.717+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:42.717+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:42.717+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:43.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:42.717+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:43.830 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:43.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:43.831 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:43.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:43.831 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:43.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:43.831 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:43.689+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:44.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:43.828+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:44.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:43.828+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:44.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:43.828+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:44.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:43.829+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:43.907+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:43.907+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:43.907+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:43.907+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:44.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:44.706+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:44.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:44.706+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:44.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:44.706+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:44.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:44.707+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:44.874+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:44.874+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:44.874+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:44.874+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:44.825+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:44.825+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:44.825+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:44.826+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:56:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:56:45.872+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 8542 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:56:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:45.836+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:45.836+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:45.836+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:45.837+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:46.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:45.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:46.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:45.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:46.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:45.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:46.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:45.897+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:45.734+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:45.734+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:45.734+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:45.735+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:47.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:46.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:47.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:46.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:47.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:46.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:46.793+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:46.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:46.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:46.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:46.891+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:46.745+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:46.745+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:47.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:46.745+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:47.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:46.745+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:47.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:47.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:47.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:47.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:47.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:47.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:47.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:47.818+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:47.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:47.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:47.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:47.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:47.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:47.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:47.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:47.926+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:47.709+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:47.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:47.709+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:47.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:47.709+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:47.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:47.710+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:48.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:48.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:48.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:48.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:48.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:48.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:48.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:48.842+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:48.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:48.663+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:48.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:48.663+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:48.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:48.663+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:48.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:48.664+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:48.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:48.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:48.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:48.929+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:49.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:49.851+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:49.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:49.851+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:49.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:49.851+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:49.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:49.852+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:49.945 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:49.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:49.945 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:49.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:49.945 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:49.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:49.946 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:49.619+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:50.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:49.942+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:50.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:49.942+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:50.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:49.942+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:50.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:49.944+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:50.846 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:50.589+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:50.846 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:50.589+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:50.846 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:50.589+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:50.846 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:50.589+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:51.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:56:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:56:50.873+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 8638 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:56:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:50.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:50.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:50.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:50.844+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:51.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:50.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:51.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:50.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:51.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:50.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:51.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:50.918+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:51.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:51.549+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:51.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:51.549+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:51.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:51.549+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:51.848 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:51.550+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:52.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:51.880+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:51.880+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:51.880+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:51.880+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:51.845+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:51.845+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:51.845+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:51.846+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:52.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:52.565+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:52.856 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:52.565+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:52.856 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:52.565+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:52.856 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:52.566+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:53.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:52.869+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:52.869+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:52.869+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:52.870+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:52.853+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:52.853+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:52.853+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:52.854+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:53.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:53.605+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:53.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:53.606+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:53.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:53.606+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:53.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:53.606+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:54.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:53.877+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:54.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:53.877+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:54.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:53.877+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:54.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:53.878+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:53.872+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:53.872+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:53.872+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:53.872+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:54.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:54.844+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:54.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:54.844+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:54.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:54.844+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:54.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:54.845+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:54.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:54.649+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:54.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:54.649+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:54.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:54.649+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:54.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:54.650+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:55.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:54.920+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:55.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:54.920+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:54.920+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:54.921+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:55.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:56:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:56:55.875+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 8730 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:56:55.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:55.849+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:55.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:55.849+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:55.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:55.849+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:55.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:55.849+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:55.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:55.884+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:55.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:55.884+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:55.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:55.884+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:55.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:55.885+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:55.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:55.682+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:55.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:55.682+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:55.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:55.682+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:55.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:55.682+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:56.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:56.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:56.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:56.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:56.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:56.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:56.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:56.879+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:56.839+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:56.839+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:56.839+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:56.840+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:56.663+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:56.663+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:56.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:56.663+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:56.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:56.663+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:57.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:57.866+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:57.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:57.866+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:57.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:57.866+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:57.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:57.867+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:57.695+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:57.695+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:57.695+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:57.696+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:57.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:57.919+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:57.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:57.919+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:57.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:57.919+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:57.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:57.919+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:58.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:58.919+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:58.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:58.919+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:58.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:58.919+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:56:58.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:58.920+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:56:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:58.683+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:58.683+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:58.683+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:58.684+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:58.831+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:58.831+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:58.831+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:58.832+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:59.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:59.795+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:59.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:59.795+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:59.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:59.795+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:56:59.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:56:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:56:59.796+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:56:59.906 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:59.653+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:59.906 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:59.653+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:59.906 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:59.653+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:56:59.907 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:56:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:56:59.653+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:59.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:59.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:59.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:56:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:56:59.904+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:00.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:00.788+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:00.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:00.788+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:00.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:00.788+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:00.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:00.789+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:00.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:00.615+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:00.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:00.615+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:00.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:00.615+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:00.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:00.616+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:01.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:57:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:57:00.876+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 8824 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:57:01.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:00.867+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:00.867+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:00.867+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:00.868+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:01.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:01.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:01.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:01.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:01.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:01.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:01.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:01.856+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:01.770+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:01.770+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:01.770+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:01.771+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:01.637+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:01.637+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:01.637+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:01.637+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:02.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:02.597+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:02.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:02.597+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:02.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:02.597+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:02.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:02.598+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:02.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:02.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:02.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:02.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:02.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:02.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:02.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:02.733+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:02.869+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:02.869+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:02.869+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:02.870+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:03.833 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:03.750+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:03.833 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:03.750+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:03.833 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:03.750+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:03.833 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:03.750+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:03.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:03.558+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:03.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:03.558+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:03.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:03.558+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:03.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:03.559+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:03.831+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:03.831+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:03.831+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:03.832+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:04.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:04.777+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:04.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:04.777+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:04.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:04.777+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:04.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:04.778+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:04.849 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:04.542+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:04.849 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:04.542+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:04.849 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:04.542+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:04.849 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:04.543+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:04.846+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:04.846+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:04.846+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:04.847+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:05.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:05.731+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:05.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:05.731+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:05.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:05.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:05.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:05.732+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:05.877 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:05.593+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:05.877 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:05.593+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:05.878 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:05.593+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:05.878 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:05.593+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:06.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:57:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:57:05.877+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 8917 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:57:06.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:05.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:05.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:05.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:05.876+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:06.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:06.850+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:06.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:06.850+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:06.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:06.850+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:06.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:06.850+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:06.768+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:06.768+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:06.768+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:06.769+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:06.640+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:06.640+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:06.640+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:06.640+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:07.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:07.828+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:07.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:07.828+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:07.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:07.828+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:07.828+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:07.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:07.792+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:07.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:07.792+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:07.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:07.792+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:07.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:07.792+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:07.601+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:07.601+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:07.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:07.601+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:07.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:07.602+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:08.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:08.802+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:08.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:08.802+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:08.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:08.802+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:08.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:08.803+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:08.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:08.758+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:08.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:08.758+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:08.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:08.758+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:08.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:08.758+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:08.598+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:08.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:08.598+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:08.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:08.598+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:08.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:08.599+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:09.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:09.762+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:09.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:09.762+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:09.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:09.762+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:09.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:09.762+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:09.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:09.771+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:09.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:09.771+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:09.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:09.771+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:09.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:09.772+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:09.602+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:09.602+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:09.602+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:09.603+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:10.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:10.744+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:10.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:10.744+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:10.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:10.744+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:10.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:10.744+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:10.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:10.739+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:10.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:10.739+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:10.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:10.739+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:10.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:10.739+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:10.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:10.554+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:10.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:10.554+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:10.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:10.554+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:10.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:10.555+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:11.184 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:57:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:57:10.879+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 9013 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:57:11.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:11.518+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:11.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:11.518+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:11.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:11.518+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:11.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:11.519+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:12.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:11.784+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:12.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:11.784+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:11.784+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:11.785+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:11.771+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:11.771+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:11.771+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:11.772+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:12.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:12.492+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:12.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:12.492+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:12.492+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:12.493+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:12.723+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:12.723+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:12.723+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:12.724+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:12.814+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:13.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:12.814+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:13.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:12.814+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:13.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:12.815+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:13.806 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:13.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:13.806 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:13.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:13.806 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:13.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:13.806 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:13.720+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:13.806 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:13.538+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:13.807 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:13.538+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:13.807 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:13.538+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:13.807 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:13.538+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:13.804+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:13.804+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:13.804+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:13.804+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:14.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:14.718+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:14.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:14.718+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:14.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:14.718+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:14.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:14.719+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:14.553+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:14.553+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:14.553+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:14.553+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:14.782+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:14.782+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:14.782+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:14.783+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:15.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:15.744+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:15.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:15.744+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:15.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:15.744+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:15.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:15.745+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:15.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:15.593+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:15.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:15.593+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:15.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:15.593+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:15.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:15.594+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:15.883 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:15.707+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:15.883 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:15.707+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:15.883 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:15.707+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:15.883 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:15.708+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:16.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:57:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:57:15.880+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 9100 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:57:16.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:16.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:16.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:16.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:16.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:16.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:16.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:16.601+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:16.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:16.701+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:16.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:16.701+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:16.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:16.701+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:16.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:16.702+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:16.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:16.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:16.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:16.712+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:17.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:17.556+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:17.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:17.556+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:17.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:17.556+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:17.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:17.556+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:17.732+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:17.732+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:17.732+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:17.733+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:17.730+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:17.730+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:17.730+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:17.731+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:18.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:18.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:18.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:18.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:18.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:18.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:18.747+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:18.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:18.756+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:18.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:18.756+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:18.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:18.756+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:18.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:18.757+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:18.550+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:18.550+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:18.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:18.550+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:18.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:18.551+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:19.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:19.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:19.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:19.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:19.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:19.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:19.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:19.593+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:19.735+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:19.735+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:19.735+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:19.736+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:19.760+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:19.760+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:19.760+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:19.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:19.761+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:20.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:20.565+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:20.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:20.565+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:20.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:20.565+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:20.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:20.566+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:21.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:57:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:57:20.881+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 9190 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:57:21.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:20.779+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:21.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:20.779+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:20.779+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:20.779+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:21.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:20.805+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:21.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:20.805+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:21.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:20.805+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:21.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:20.806+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:21.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:21.781+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:21.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:21.781+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:21.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:21.781+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:21.782+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:21.825+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:21.825+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:21.825+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:21.825+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:21.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:21.606+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:21.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:21.606+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:21.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:21.606+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:21.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:21.607+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:22.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:22.770+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:22.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:22.770+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:22.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:22.770+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:22.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:22.771+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:22.877 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:22.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:22.877 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:22.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:22.877 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:22.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:22.877 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:22.577+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:23.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:22.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:23.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:22.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:23.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:22.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:23.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:22.875+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:23.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:23.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:23.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:23.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:23.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:23.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:23.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:23.601+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:24.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:23.871+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:24.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:23.871+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:24.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:23.871+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:24.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:23.871+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:24.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:23.751+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:24.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:23.751+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:24.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:23.751+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:24.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:23.752+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:24.828 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:24.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:24.829 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:24.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:24.829 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:24.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:24.829 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:24.715+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:24.829 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:24.552+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:24.830 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:24.552+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:24.830 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:24.552+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:24.830 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:24.553+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:24.826+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:24.826+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:24.826+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:24.827+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:25.817 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:25.764+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:25.817 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:25.764+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:25.817 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:25.764+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:25.817 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:25.765+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:25.818 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:25.556+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:25.818 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:25.556+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:25.818 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:25.556+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:25.818 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:25.556+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:26.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:57:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:57:25.881+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 9275 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:57:26.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:25.814+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:26.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:25.814+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:25.814+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:25.815+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:26.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:26.537+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:26.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:26.537+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:26.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:26.537+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:26.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:26.537+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:27.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:26.775+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:27.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:26.775+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:26.775+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:26.775+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:26.779+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:26.779+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:26.779+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:26.780+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:27.815 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:27.743+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:27.815 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:27.743+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:27.815 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:27.743+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:27.815 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:27.743+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:27.816 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:27.508+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:27.816 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:27.508+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:27.816 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:27.508+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:27.816 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:27.509+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:28.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:27.812+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:28.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:27.812+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:28.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:27.812+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:28.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:27.813+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:28.797 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:28.509+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:28.797 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:28.509+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:28.797 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:28.509+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:28.797 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:28.510+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:28.798 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:28.740+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:28.798 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:28.740+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:28.798 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:28.740+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:28.798 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:28.740+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:29.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:28.794+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:29.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:28.794+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:29.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:28.794+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:29.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:28.795+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:29.781+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:29.781+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:29.781+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:29.781+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:29.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:29.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:29.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:29.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:29.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:29.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:29.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:29.753+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:29.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:29.552+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:29.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:29.552+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:29.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:29.552+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:29.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:29.552+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:30.886 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:30.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:30.886 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:30.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:30.886 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:30.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:30.886 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:30.736+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:30.887 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:30.759+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:30.887 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:30.759+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:30.887 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:30.759+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:30.887 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:30.759+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:30.887 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:30.572+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:30.887 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:30.572+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:30.887 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:30.572+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:30.887 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:30.573+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:31.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:57:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:57:30.884+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 9363 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:57:31.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:31.599+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:31.599+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:31.599+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:31.600+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:31.759+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:31.759+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:31.759+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:31.760+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:31.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:31.728+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:31.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:31.728+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:31.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:31.728+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:31.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:31.729+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:32.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:32.695+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:32.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:32.695+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:32.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:32.695+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:32.696+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:32.582+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:32.582+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:32.582+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:32.583+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:32.793+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:32.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:32.793+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:32.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:32.793+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:32.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:32.793+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:33.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:33.752+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:33.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:33.752+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:33.752+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:33.752+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:33.659+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:33.659+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:33.659+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:33.660+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:33.555+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:33.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:33.555+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:33.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:33.555+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:33.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:33.556+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:34.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:34.731+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:34.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:34.731+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:34.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:34.731+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:34.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:34.731+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:34.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:34.632+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:34.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:34.632+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:34.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:34.632+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:34.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:34.633+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:34.517+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:34.517+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:34.517+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:34.518+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:35.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:35.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:35.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:35.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:35.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:35.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:35.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:35.502+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:35.760 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:35.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:35.760 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:35.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:35.760 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:35.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:35.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:35.747+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:35.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:35.648+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:35.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:35.648+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:35.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:35.648+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:35.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:35.649+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:36.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:57:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:57:35.885+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 9450 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:57:36.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:36.504+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:36.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:36.504+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:36.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:36.504+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:36.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:36.505+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:36.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:36.752+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:36.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:36.752+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:36.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:36.752+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:36.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:36.753+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:36.667+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:36.667+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:36.667+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:36.668+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:37.823 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:37.635+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:37.823 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:37.635+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:37.823 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:37.635+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:37.823 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:37.636+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:37.824 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:37.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:37.824 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:37.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:37.824 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:37.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:37.824 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:37.752+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:37.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:37.509+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:37.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:37.509+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:37.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:37.509+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:37.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:37.510+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:38.833 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:38.554+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:38.833 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:38.554+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:38.833 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:38.554+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:38.833 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:38.555+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:38.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:38.741+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:38.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:38.741+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:38.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:38.741+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:38.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:38.741+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:38.834 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:38.627+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:38.834 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:38.627+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:38.835 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:38.627+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:38.835 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:38.628+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:39.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:39.718+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:39.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:39.718+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:39.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:39.718+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:39.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:39.718+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:39.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:39.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:39.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:39.597+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:39.672+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:39.672+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:39.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:39.672+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:39.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:39.672+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:40.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:40.629+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:40.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:40.629+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:40.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:40.629+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:40.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:40.630+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:40.889 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:40.766+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:40.889 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:40.766+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:40.889 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:40.766+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:40.889 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:40.766+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:40.890 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:40.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:40.890 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:40.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:40.890 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:40.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:40.890 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:40.581+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:57:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:57:40.886+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 9543 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:57:41.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:41.801+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:41.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:41.801+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:41.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:41.801+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:41.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:41.802+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:41.614+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:41.614+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:41.614+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:41.615+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:41.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:41.621+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:41.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:41.621+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:41.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:41.621+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:41.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:41.622+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:42.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:42.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:42.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:42.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:42.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:42.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:42.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:42.782+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:42.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:42.589+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:42.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:42.589+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:42.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:42.589+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:42.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:42.590+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:42.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:42.625+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:42.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:42.625+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:42.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:42.625+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:42.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:42.626+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:43.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:43.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:43.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:43.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:43.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:43.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:43.587+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:43.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:43.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:43.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:43.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:43.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:43.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:43.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:43.741+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:43.619+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:43.619+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:43.619+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:43.620+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:44.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:44.598+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:44.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:44.598+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:44.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:44.598+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:44.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:44.599+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:44.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:44.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:44.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:44.661+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:44.735+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:44.735+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:44.735+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:44.736+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:45.890 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:45.691+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:45.890 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:45.691+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:45.890 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:45.691+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:45.890 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:45.692+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:45.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:45.636+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:45.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:45.636+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:45.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:45.636+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:45.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:45.637+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:45.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:45.610+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:45.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:45.610+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:45.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:45.610+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:45.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:45.611+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:57:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:57:45.888+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 9632 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:57:46.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:46.594+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:46.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:46.594+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:46.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:46.594+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:46.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:46.595+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:46.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:46.591+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:46.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:46.591+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:46.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:46.591+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:46.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:46.591+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:46.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:46.700+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:46.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:46.700+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:46.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:46.700+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:46.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:46.701+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:47.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:47.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:47.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:47.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:47.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:47.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:47.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:47.747+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:47.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:47.545+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:47.867 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:47.545+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:47.867 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:47.545+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:47.867 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:47.545+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:47.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:47.584+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:47.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:47.584+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:47.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:47.584+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:47.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:47.585+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:48.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:48.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:48.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:48.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:48.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:48.754+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:48.524+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:48.524+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:48.524+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:48.525+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:48.569+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:48.569+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:48.569+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:48.570+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:49.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:49.553+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:49.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:49.553+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:49.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:49.553+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:49.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:49.553+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:49.523+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:49.523+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:49.523+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:49.524+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:49.761+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:49.761+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:49.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:49.761+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:49.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:49.762+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:50.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:50.725+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:50.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:50.725+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:50.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:50.725+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:50.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:50.726+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:50.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:50.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:50.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:50.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:50.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:50.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:50.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:50.588+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:50.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:50.545+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:50.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:50.545+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:50.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:50.545+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:50.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:50.546+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:51.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:57:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:57:50.889+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 9727 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:57:51.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:51.594+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:51.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:51.594+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:51.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:51.594+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:51.595+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:51.724+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:51.724+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:51.724+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:51.725+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:51.618+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:51.618+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:51.618+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:51.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:51.618+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:52.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:52.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:52.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:52.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:52.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:52.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:52.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:52.604+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:52.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:52.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:52.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:52.592+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:52.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:52.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:52.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:52.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:52.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:52.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:52.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:52.753+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:53.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:53.557+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:53.557+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:53.557+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:53.558+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:53.730+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:53.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:53.730+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:53.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:53.730+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:53.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:53.731+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:53.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:53.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:53.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:53.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:53.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:53.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:53.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:53.618+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:54.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:54.584+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:54.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:54.584+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:54.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:54.584+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:54.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:54.585+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:54.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:54.759+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:54.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:54.759+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:54.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:54.759+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:54.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:54.759+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:54.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:54.590+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:54.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:54.590+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:54.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:54.590+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:54.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:54.591+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:55.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:55.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:55.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:55.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:55.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:55.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:55.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:55.581+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:55.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:55.784+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:55.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:55.784+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:55.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:55.784+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:55.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:55.784+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:55.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:55.535+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:55.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:55.535+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:55.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:55.535+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:55.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:55.536+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:56.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:57:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:57:55.890+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 9818 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:57:56.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:56.548+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:56.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:56.548+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:56.548+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:56.549+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:56.553+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:56.553+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:56.553+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:56.554+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:56.784+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:56.784+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:57.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:56.784+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:57.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:56.784+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:57.832 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:57.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:57.832 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:57.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:57.834 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:57.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:57.834 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:57.554+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:57.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:57.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:57.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:57.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:57.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:57.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:57.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:57.581+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:57.830+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:57.830+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:57.830+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:57.831+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:58.828+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:58.828+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:58.828+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:57:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:58.829+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:57:58.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:58.600+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:58.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:58.600+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:58.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:58.600+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:58.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:58.601+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:58.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:58.606+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:58.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:58.606+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:58.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:58.606+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:58.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:58.607+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:59.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:59.619+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:59.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:59.619+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:59.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:59.619+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:57:59.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:57:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:57:59.620+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:57:59.884 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:59.637+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:59.884 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:59.637+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:59.884 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:59.637+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:57:59.884 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:57:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:57:59.637+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:59.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:59.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:59.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:57:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:57:59.879+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:00.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:00.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:00.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:00.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:00.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:00.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:00.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:00.581+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:00.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:00.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:00.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:00.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:00.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:00.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:00.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:00.597+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:01.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:58:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:58:00.892+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 9912 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:58:01.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:00.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:01.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:00.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:00.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:00.880+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:01.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:01.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:01.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:01.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:01.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:01.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:01.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:01.599+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:01.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:01.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:01.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:01.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:01.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:01.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:01.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:01.572+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:02.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:01.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:02.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:01.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:02.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:01.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:02.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:01.915+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:02.893 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:02.553+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:02.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:02.553+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:02.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:02.553+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:02.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:02.554+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:02.894 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:02.543+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:02.894 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:02.543+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:02.895 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:02.543+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:02.895 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:02.544+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:02.891+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:02.891+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:02.891+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:02.892+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:03.886 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:03.583+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:03.886 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:03.583+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:03.886 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:03.583+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:03.886 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:03.583+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:03.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:03.582+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:03.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:03.582+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:03.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:03.582+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:03.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:03.583+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:03.884+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:03.884+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:03.884+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:03.884+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:04.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:04.552+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:04.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:04.552+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:04.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:04.552+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:04.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:04.552+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:04.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:04.546+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:04.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:04.546+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:04.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:04.546+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:04.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:04.547+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:04.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:04.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:04.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:04.841+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:05.821 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:05.540+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:05.822 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:05.540+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:05.822 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:05.540+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:05.822 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:05.541+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:05.822 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:05.532+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:05.822 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:05.532+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:05.822 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:05.532+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:05.822 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:05.533+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:06.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:58:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:58:05.893+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 10004 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:58:06.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:05.819+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:06.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:05.819+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:05.819+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:05.820+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:06.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:06.527+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:06.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:06.527+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:06.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:06.527+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:06.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:06.528+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:06.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:06.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:06.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:06.506+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:06.845+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:06.845+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:06.845+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:06.845+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:07.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:07.529+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:07.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:07.529+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:07.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:07.529+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:07.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:07.529+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:07.463+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:07.463+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:07.463+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:07.463+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:08.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:07.842+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:08.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:07.842+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:07.842+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:07.842+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:08.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:08.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:08.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:08.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:08.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:08.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:08.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:08.416+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:08.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:08.543+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:08.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:08.543+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:08.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:08.543+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:08.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:08.544+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:09.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:08.884+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:09.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:08.884+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:09.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:08.884+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:08.885+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:09.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:09.500+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:09.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:09.500+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:09.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:09.500+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:09.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:09.501+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:09.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:09.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:09.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:09.424+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:10.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:09.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:10.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:09.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:10.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:09.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:10.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:09.848+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:10.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:10.454+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:10.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:10.454+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:10.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:10.454+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:10.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:10.455+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:10.486+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:10.486+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:10.486+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:10.487+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:11.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:58:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:58:10.894+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 10099 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:58:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:10.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:10.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:10.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:10.834+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:11.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:11.492+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:11.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:11.492+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:11.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:11.492+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:11.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:11.493+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:11.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:11.517+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:11.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:11.517+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:11.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:11.517+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:11.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:11.517+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:11.873+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:11.873+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:11.873+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:11.873+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:12.848 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:12.501+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:12.848 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:12.501+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:12.848 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:12.501+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:12.849 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:12.501+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:12.849 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:12.498+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:12.849 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:12.498+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:12.849 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:12.498+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:12.849 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:12.499+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:12.845+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:12.845+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:12.845+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:12.846+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:13.879 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:13.501+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:13.879 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:13.501+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:13.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:13.501+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:13.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:13.502+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:13.880 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:13.499+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:13.880 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:13.499+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:13.880 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:13.499+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:13.880 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:13.500+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:13.876+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:13.876+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:13.876+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:13.877+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:14.839 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:14.504+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:14.840 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:14.504+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:14.840 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:14.504+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:14.840 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:14.505+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:14.840 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:14.538+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:14.840 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:14.538+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:14.840 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:14.538+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:14.840 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:14.538+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:15.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:14.836+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:15.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:14.836+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:15.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:14.836+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:15.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:14.837+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:15.823 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:15.474+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:15.824 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:15.474+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:15.824 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:15.474+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:15.824 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:15.474+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:15.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:15.559+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:15.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:15.559+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:15.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:15.559+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:15.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:15.560+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:16.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:58:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:58:15.895+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 10191 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:58:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:15.820+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:15.820+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:15.820+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:15.821+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:16.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:16.511+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:16.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:16.511+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:16.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:16.511+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:16.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:16.512+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:16.451+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:16.451+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:16.451+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:16.451+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:17.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:16.775+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:17.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:16.775+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:16.775+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:16.776+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:17.748 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:17.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:17.748 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:17.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:17.748 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:17.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:17.748 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:17.481+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:17.748 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:17.536+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:17.748 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:17.536+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:17.748 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:17.536+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:17.748 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:17.537+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:17.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:17.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:17.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:17.746+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:18.768 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:18.705+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:18.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:18.705+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:18.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:18.705+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:18.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:18.706+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:18.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:18.550+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:18.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:18.550+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:18.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:18.550+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:18.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:18.550+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:18.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:18.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:18.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:18.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:18.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:18.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:18.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:18.474+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:19.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:19.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:19.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:19.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:19.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:19.474+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:19.662+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:19.662+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:19.662+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:19.663+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:19.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:19.570+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:19.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:19.570+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:19.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:19.570+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:19.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:19.571+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:20.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:20.653+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:20.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:20.653+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:20.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:20.653+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:20.653+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:20.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:20.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:20.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:20.461+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:20.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:20.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:20.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:20.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:20.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:20.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:20.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:20.589+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:21.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:58:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:58:20.897+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 10286 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:58:21.550 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:21.412+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:21.550 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:21.412+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:21.550 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:21.412+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:21.550 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:21.413+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:21.840 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:21.548+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:21.840 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:21.548+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:21.840 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:21.548+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:21.840 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:21.548+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:21.841 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:21.643+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:21.841 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:21.643+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:21.841 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:21.643+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:21.841 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:21.644+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:22.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:22.394+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:22.678 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:22.394+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:22.678 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:22.394+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:22.678 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:22.395+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:22.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:22.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:22.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:22.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:22.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:22.502+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:22.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:22.502+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:22.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:22.675+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:22.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:22.675+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:22.675+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:22.676+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:23.655 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:23.346+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:23.656 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:23.346+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:23.656 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:23.346+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:23.656 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:23.347+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:23.656 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:23.514+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:23.656 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:23.514+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:23.656 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:23.514+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:23.657 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:23.515+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:23.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:23.653+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:23.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:23.653+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:23.653+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:23.654+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:24.680 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:24.382+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:24.680 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:24.382+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:24.680 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:24.382+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:24.681 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:24.382+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:24.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:24.511+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:24.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:24.511+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:24.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:24.511+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:24.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:24.512+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:24.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:24.678+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:24.678+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:24.678+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:24.679+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:25.683 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:25.357+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:25.683 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:25.357+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:25.683 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:25.357+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:25.683 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:25.358+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:25.683 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:25.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:25.683 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:25.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:25.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:25.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:25.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:25.561+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:25.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:58:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:58:25.899+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 10378 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:58:25.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:25.681+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:25.681+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:25.681+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:25.681+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:26.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:26.313+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:26.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:26.313+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:26.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:26.313+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:26.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:26.313+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:26.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:26.526+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:26.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:26.526+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:26.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:26.526+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:26.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:26.527+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:27.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:26.714+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:27.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:26.714+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:27.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:26.714+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:26.715+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:27.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:27.567+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:27.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:27.567+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:27.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:27.567+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:27.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:27.567+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:27.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:27.333+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:27.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:27.333+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:27.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:27.333+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:27.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:27.333+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:27.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:27.711+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:27.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:27.711+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:27.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:27.711+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:27.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:27.711+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:28.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:28.367+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:28.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:28.367+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:28.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:28.367+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:28.367+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:28.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:28.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:28.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:28.587+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:29.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:28.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:29.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:28.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:29.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:28.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:29.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:28.754+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:29.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:29.626+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:29.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:29.626+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:29.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:29.626+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:29.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:29.626+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:29.406+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:29.406+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:29.406+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:29.407+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:30.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:29.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:30.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:29.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:30.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:29.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:30.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:29.737+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:30.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:30.664+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:30.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:30.664+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:30.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:30.664+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:30.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:30.664+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:30.446+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:30.446+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:30.446+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:30.447+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:31.056 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:58:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:58:30.900+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 10474 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:58:31.056 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:30.737+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:31.056 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:30.737+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:31.056 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:30.737+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:31.056 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:30.738+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:31.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:31.401+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:31.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:31.401+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:31.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:31.401+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:31.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:31.401+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:31.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:31.655+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:31.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:31.655+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:31.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:31.655+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:31.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:31.655+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:31.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:31.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:31.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:31.741+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:32.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:32.364+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:32.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:32.364+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:32.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:32.364+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:32.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:32.365+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:32.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:32.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:32.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:32.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:32.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:32.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:32.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:32.780+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:32.644+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:32.644+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:32.644+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:32.645+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:33.623 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:33.354+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:33.623 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:33.354+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:33.623 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:33.354+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:33.623 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:33.355+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:33.903 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:33.621+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:33.903 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:33.621+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:33.903 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:33.621+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:33.903 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:33.621+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:33.903 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:33.824+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:33.903 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:33.824+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:33.903 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:33.824+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:33.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:33.825+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:34.602 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:34.347+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:34.602 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:34.347+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:34.602 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:34.347+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:34.602 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:34.348+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:34.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:34.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:34.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:34.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:34.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:34.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:34.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:34.601+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:34.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:34.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:34.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:34.822+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:35.606 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:35.312+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:35.606 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:35.312+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:35.606 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:35.312+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:35.606 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:35.313+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:35.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:35.604+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:35.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:35.604+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:35.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:35.604+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:35.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:35.605+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:36.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:58:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:58:35.901+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 10566 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:58:36.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:35.856+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:36.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:35.856+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:36.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:35.856+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:35.856+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:36.621 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:36.303+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:36.622 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:36.303+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:36.622 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:36.303+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:36.622 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:36.304+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:36.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:36.620+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:36.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:36.620+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:36.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:36.620+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:36.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:36.620+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:37.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:36.881+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:37.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:36.881+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:37.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:36.881+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:37.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:36.882+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:37.666 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:37.349+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:37.666 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:37.349+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:37.666 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:37.349+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:37.666 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:37.350+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:37.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:37.664+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:37.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:37.664+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:37.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:37.664+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:37.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:37.665+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:38.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:37.920+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:38.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:37.920+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:38.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:37.920+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:38.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:37.921+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:38.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:38.368+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:38.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:38.368+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:38.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:38.368+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:38.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:38.368+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:38.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:38.635+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:38.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:38.635+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:38.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:38.635+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:38.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:38.636+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:38.913+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:38.913+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:38.913+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:38.914+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:39.667 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:39.400+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:39.667 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:39.400+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:39.667 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:39.400+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:39.667 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:39.400+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:39.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:39.665+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:39.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:39.665+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:39.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:39.665+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:39.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:39.666+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:40.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:39.959+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:40.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:39.959+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:40.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:39.959+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:40.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:39.960+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:40.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:40.402+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:40.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:40.402+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:40.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:40.402+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:40.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:40.403+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:40.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:58:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:58:40.903+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 10661 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:58:40.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:40.911+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:40.911+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:40.911+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:40.913+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:40.700+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:40.700+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:40.700+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:40.701+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:41.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:41.357+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:41.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:41.357+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:41.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:41.357+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:41.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:41.358+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:42.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:41.870+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:42.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:41.870+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:42.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:41.870+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:42.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:41.871+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:41.739+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:41.739+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:41.739+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:41.739+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:42.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:42.372+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:42.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:42.372+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:42.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:42.372+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:42.372+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:43.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:42.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:43.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:42.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:43.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:42.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:42.856+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:42.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:42.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:42.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:42.768+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:43.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:43.342+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:43.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:43.342+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:43.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:43.342+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:43.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:43.342+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:44.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:43.837+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:44.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:43.837+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:44.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:43.837+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:44.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:43.838+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:44.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:43.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:44.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:43.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:44.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:43.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:44.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:43.773+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:44.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:44.309+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:44.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:44.309+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:44.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:44.309+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:44.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:44.310+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:45.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:44.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:45.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:44.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:45.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:44.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:45.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:44.845+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:44.747+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:44.747+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:44.747+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:44.748+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:45.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:45.328+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:45.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:45.328+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:45.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:45.328+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:45.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:45.329+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:58:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:58:45.904+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 10754 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:58:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:45.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:45.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:45.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:45.887+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:45.766+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:45.766+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:45.766+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:45.767+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:46.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:46.362+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:46.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:46.362+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:46.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:46.362+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:46.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:46.363+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:47.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:46.890+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:47.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:46.890+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:47.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:46.890+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:47.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:46.891+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:46.723+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:46.723+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:46.723+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:46.724+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:47.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:47.403+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:47.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:47.403+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:47.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:47.403+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:47.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:47.404+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:48.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:47.724+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:48.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:47.724+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:48.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:47.724+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:48.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:47.724+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:47.890+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:47.890+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:47.890+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:47.890+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:48.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:48.417+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:48.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:48.417+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:48.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:48.417+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:48.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:48.418+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:49.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:48.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:49.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:48.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:49.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:48.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:49.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:48.863+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:49.194 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:48.769+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:49.194 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:48.769+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:49.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:48.769+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:49.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:48.770+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:49.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:49.399+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:49.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:49.399+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:49.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:49.399+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:49.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:49.399+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:50.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:49.909+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:50.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:49.909+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:50.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:49.909+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:50.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:49.909+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:50.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:49.755+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:50.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:49.755+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:50.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:49.755+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:50.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:49.756+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:50.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:50.399+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:50.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:50.399+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:50.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:50.399+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:50.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:50.399+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:51.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:58:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:58:50.905+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 10850 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:58:51.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:50.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:50.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:50.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:50.791+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:50.867+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:50.867+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:50.867+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:50.867+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:51.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:51.409+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:51.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:51.409+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:51.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:51.409+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:51.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:51.409+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:52.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:51.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:52.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:51.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:52.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:51.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:51.883+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:51.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:51.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:51.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:51.838+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:52.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:52.457+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:52.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:52.457+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:52.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:52.457+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:52.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:52.457+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:52.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:52.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:52.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:52.825+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:52.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:52.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:52.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:52.879+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:53.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:53.430+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:53.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:53.430+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:53.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:53.430+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:53.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:53.431+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:54.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:53.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:54.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:53.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:54.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:53.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:54.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:53.863+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:53.813+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:53.813+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:53.813+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:53.813+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:54.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:54.398+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:54.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:54.398+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:54.398+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:54.398+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:55.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:54.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:55.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:54.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:55.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:54.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:55.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:54.872+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:54.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:54.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:54.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:54.817+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:55.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:55.375+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:55.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:55.375+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:55.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:55.375+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:55.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:55.376+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:55.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:55.867+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:55.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:55.867+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:55.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:55.867+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:55.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:55.867+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:55.910 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:55.852+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:55.910 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:55.852+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:55.910 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:55.852+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:55.910 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:55.853+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:56.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:58:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:58:55.907+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 10942 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:58:56.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:56.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:56.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:56.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:56.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:56.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:56.416+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:56.847+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:56.847+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:56.847+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:57.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:56.848+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:56.888+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:56.888+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:56.888+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:56.888+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:57.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:57.376+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:57.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:57.376+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:57.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:57.376+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:57.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:57.376+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:57.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:57.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:57.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:57.872+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:57.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:57.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:57.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:57.842+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:58.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:58.418+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:58.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:58.418+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:58.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:58.418+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:58.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:58.419+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:59.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:58.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:59.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:58.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:59.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:58.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:58:59.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:58.870+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:58:59.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:58.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:59.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:58.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:59.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:58.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:58:59.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:58.873+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:58:59.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:59.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:59.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:59.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:58:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:58:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:58:59.438+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:00.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:59.841+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:00.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:59.841+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:00.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:59.841+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:00.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:58:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:58:59.842+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:00.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:59.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:00.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:59.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:00.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:59.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:00.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:58:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:58:59.883+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:00.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:00.480+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:00.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:00.480+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:00.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:00.480+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:00.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:00.481+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:01.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:59:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:59:00.907+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 11037 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:59:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:00.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:00.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:00.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:00.840+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:00.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:00.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:00.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:00.879+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:01.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:01.508+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:01.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:01.508+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:01.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:01.508+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:01.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:01.508+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:02.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:01.908+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:02.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:01.908+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:01.908+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:01.909+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:01.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:01.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:01.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:01.834+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:02.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:02.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:02.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:02.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:02.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:02.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:02.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:02.511+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:02.882+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:02.883+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:02.883+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:02.883+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:02.868+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:02.868+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:02.868+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:02.869+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:03.826 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:03.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:03.826 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:03.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:03.826 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:03.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:03.826 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:03.531+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:03.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:03.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:03.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:03.838+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:03.824+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:03.824+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:03.824+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:03.825+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:04.805 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:04.518+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:04.805 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:04.518+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:04.805 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:04.518+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:04.805 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:04.518+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:04.803+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:04.803+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:04.803+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:04.804+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:04.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:04.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:04.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:04.858+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:05.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:05.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:05.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:05.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:05.784 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:05.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:05.784 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:05.490+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:06.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:59:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:59:05.910+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 11128 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:59:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:05.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:05.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:05.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:05.900+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:05.781+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:05.781+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:05.781+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:05.782+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:06.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:06.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:06.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:06.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:06.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:06.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:06.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:06.515+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:06.769+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:06.769+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:06.769+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:06.770+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:06.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:06.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:06.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:06.869+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:07.739 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:07.487+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:07.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:07.487+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:07.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:07.487+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:07.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:07.488+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:07.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:07.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:07.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:07.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:07.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:07.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:07.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:07.737+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:08.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:07.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:08.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:07.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:08.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:07.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:08.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:07.831+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:08.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:08.480+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:08.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:08.480+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:08.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:08.480+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:08.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:08.481+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:09.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:08.786+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:09.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:08.786+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:09.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:08.786+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:09.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:08.786+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:08.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:08.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:08.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:08.809+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:09.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:09.436+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:09.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:09.436+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:09.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:09.436+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:09.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:09.437+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:10.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:09.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:10.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:09.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:10.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:09.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:10.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:09.796+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:10.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:09.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:10.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:09.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:10.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:09.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:10.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:09.784+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:10.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:10.485+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:10.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:10.485+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:10.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:10.485+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:10.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:10.486+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:11.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:59:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:59:10.911+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 11223 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:59:11.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:10.830+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:11.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:10.830+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:10.830+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:10.830+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:10.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:10.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:10.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:10.821+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:11.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:11.450+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:11.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:11.450+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:11.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:11.450+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:11.451+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:11.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:11.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:11.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:11.880+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:11.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:11.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:11.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:11.773+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:12.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:12.472+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:12.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:12.472+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:12.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:12.472+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:12.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:12.473+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:12.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:12.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:12.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:12.856+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:12.760+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:12.760+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:12.760+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:12.760+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:13.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:13.517+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:13.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:13.517+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:13.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:13.517+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:13.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:13.518+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:13.904 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:13.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:13.904 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:13.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:13.904 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:13.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:13.904 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:13.749+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:13.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:13.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:13.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:13.902+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:14.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:14.528+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:14.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:14.528+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:14.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:14.528+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:14.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:14.529+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:15.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:14.893+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:15.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:14.893+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:15.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:14.893+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:15.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:14.894+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:14.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:14.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:14.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:14.795+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:15.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:15.499+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:15.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:15.499+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:15.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:15.499+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:15.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:15.500+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:15.915 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:15.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:15.915 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:15.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:15.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:15.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:15.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:15.749+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:16.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:59:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:59:15.913+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 11314 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:59:16.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:15.922+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:16.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:15.922+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:15.922+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:15.923+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:16.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:16.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:16.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:16.507+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:17.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:16.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:17.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:16.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:17.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:16.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:17.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:16.958+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:17.137 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:16.770+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:17.137 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:16.770+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:17.137 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:16.770+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:17.137 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:16.771+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:17.800 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:17.512+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:17.801 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:17.512+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:17.801 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:17.512+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:17.801 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:17.513+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:17.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:17.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:17.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:17.931+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:17.798+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:17.798+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:17.798+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:17.799+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:18.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:18.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:18.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:18.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:18.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:18.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:18.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:18.563+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:18.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:18.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:18.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:18.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:18.951 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:18.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:18.951 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:18.753+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:19.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:18.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:19.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:18.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:19.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:18.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:19.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:18.948+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:19.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:19.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:19.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:19.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:19.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:19.579+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:19.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:19.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:19.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:19.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:19.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:19.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:19.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:19.751+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:20.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:19.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:20.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:19.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:20.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:19.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:20.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:19.998+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:20.916 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:20.576+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:20.916 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:20.576+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:20.916 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:20.576+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:20.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:20.577+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:20.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:20.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:20.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:20.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:20.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:20.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:20.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:20.727+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:21.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:59:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:59:20.914+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 11407 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:59:21.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:21.008+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:21.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:21.008+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:21.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:21.008+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:21.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:21.009+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:21.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:21.697+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:21.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:21.697+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:21.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:21.697+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:21.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:21.698+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:21.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:21.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:21.603+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:21.604+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:22.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:22.028+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:22.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:22.028+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:22.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:22.028+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:22.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:22.028+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:22.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:22.636+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:22.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:22.636+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:22.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:22.636+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:22.637+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:22.675+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:22.675+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:22.675+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:22.676+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:23.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:23.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:23.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:23.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:23.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:23.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:23.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:23.063+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:23.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:23.618+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:23.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:23.618+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:23.618+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:23.618+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:23.637+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:23.637+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:23.637+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:23.638+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:24.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:24.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:24.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:24.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:24.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:24.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:24.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:24.061+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:24.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:24.658+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:24.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:24.658+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:24.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:24.658+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:24.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:24.658+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:24.652+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:24.652+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:24.652+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:24.653+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:25.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:25.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:25.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:25.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:25.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:25.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:25.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:25.025+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:25.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:25.606+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:25.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:25.606+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:25.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:25.606+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:25.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:25.606+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:25.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:25.621+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:25.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:25.621+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:25.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:25.621+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:25.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:25.622+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:26.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:59:25.915+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 11499 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:59:26.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:25.978+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:26.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:25.978+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:26.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:25.978+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:25.979+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:26.615 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:26.582+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:26.615 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:26.582+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:26.615 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:26.582+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:26.615 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:26.583+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:26.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:26.613+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:26.613+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:26.613+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:26.613+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:27.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:27.028+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:27.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:27.028+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:27.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:27.028+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:27.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:27.029+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:27.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:27.617+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:27.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:27.617+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:27.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:27.617+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:27.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:27.618+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:27.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:27.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:27.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:27.597+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:28.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:28.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:28.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:28.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:28.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:28.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:28.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:28.002+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:28.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:28.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:28.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:28.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:28.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:28.597+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:28.631+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:28.631+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:28.631+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:28.632+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:29.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:29.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:29.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:29.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:29.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:29.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:29.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:29.023+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:29.921 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:29.667+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:29.921 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:29.667+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:29.921 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:29.667+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:29.921 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:29.667+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:29.921 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:29.579+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:29.921 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:29.579+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:29.921 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:29.579+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:29.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:29.579+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:30.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:30.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:30.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:30.039+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:30.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:30.700+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:30.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:30.700+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:30.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:30.700+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:30.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:30.701+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:30.920 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:30.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:30.920 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:30.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:30.920 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:30.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:30.920 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:30.562+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:31.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:59:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:59:30.917+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 11594 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:59:31.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:31.056+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:31.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:31.056+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:31.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:31.056+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:31.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:31.057+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:31.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:31.602+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:31.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:31.602+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:31.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:31.602+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:31.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:31.603+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:31.708+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:31.709+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:31.709+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:31.709+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:32.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:32.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:32.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:32.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:32.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:32.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:32.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:32.017+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:32.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:32.646+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:32.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:32.646+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:32.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:32.646+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:32.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:32.647+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:32.724+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:32.724+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:32.724+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:32.725+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:33.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:33.048+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:33.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:33.048+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:33.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:33.048+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:33.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:33.049+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:33.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:33.687+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:33.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:33.687+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:33.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:33.687+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:33.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:33.687+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:33.765+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:33.765+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:33.765+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:33.766+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:34.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:34.094+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:34.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:34.094+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:34.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:34.094+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:34.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:34.094+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:35.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:34.751+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:35.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:34.751+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:35.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:34.751+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:35.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:34.752+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:35.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:34.716+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:35.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:34.716+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:35.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:34.716+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:35.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:34.716+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:35.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:35.123+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:35.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:35.123+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:35.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:35.123+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:35.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:35.123+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:35.921 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:35.704+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:35.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:35.704+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:35.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:35.704+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:35.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:35.705+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:35.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:35.716+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:35.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:35.716+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:35.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:35.716+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:35.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:35.717+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:36.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:59:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:59:35.918+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 11685 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:59:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:36.118+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:36.118+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:36.118+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:36.119+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:37.089 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:36.733+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:37.089 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:36.733+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:37.089 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:36.733+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:37.089 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:36.734+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:37.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:36.724+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:37.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:36.724+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:37.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:36.724+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:37.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:36.725+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:37.086+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:37.086+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:37.087+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:37.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:37.087+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:37.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:37.696+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:37.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:37.696+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:37.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:37.696+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:37.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:37.697+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:37.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:37.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:37.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:37.728+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:38.130+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:38.130+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:38.130+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:38.131+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:38.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:38.709+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:38.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:38.709+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:38.709+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:38.710+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:38.763+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:38.763+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:38.763+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:38.763+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:39.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:39.090+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:39.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:39.090+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:39.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:39.090+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:39.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:39.090+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:39.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:39.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:39.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:39.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:39.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:39.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:39.688+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:39.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:39.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:39.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:39.753+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:40.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:40.047+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:40.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:40.047+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:40.047+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:40.047+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:40.931 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:59:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:59:40.920+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 11779 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:59:40.931 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:40.679+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:40.931 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:40.679+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:40.931 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:40.679+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:40.931 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:40.679+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:40.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:40.758+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:40.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:40.758+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:40.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:40.758+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:40.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:40.759+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:41.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:41.031+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:41.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:41.031+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:41.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:41.031+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:41.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:41.031+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:41.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:41.697+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:41.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:41.697+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:41.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:41.697+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:41.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:41.698+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:41.754+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:41.754+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:41.754+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:41.754+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:42.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:41.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:42.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:41.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:42.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:41.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:42.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:41.985+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:42.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:42.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:42.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:42.715+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:42.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:42.715+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:42.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:42.715+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:42.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:42.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:42.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:42.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:42.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:42.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:42.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:42.791+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:43.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:42.984+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:43.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:42.984+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:43.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:42.984+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:43.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:42.984+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:43.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:43.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:43.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:43.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:43.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:43.693+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:43.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:43.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:43.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:43.768+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:44.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:44.015+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:44.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:44.015+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:44.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:44.015+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:44.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:44.016+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:44.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:44.695+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:44.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:44.695+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:44.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:44.695+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:44.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:44.696+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:44.814+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:44.814+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:44.814+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:44.814+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:45.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:45.056+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:45.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:45.056+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:45.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:45.056+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:45.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:45.057+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:45.935 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:59:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:59:45.920+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 11871 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:59:45.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:45.682+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:45.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:45.682+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:45.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:45.682+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:45.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:45.683+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:45.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:45.789+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:45.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:45.789+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:45.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:45.789+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:45.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:45.789+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:46.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:46.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:46.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:46.067+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:46.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:46.759+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:46.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:46.759+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:46.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:46.759+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:46.761+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:46.636+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:46.636+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:46.636+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:46.637+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:47.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:47.057+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:47.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:47.057+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:47.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:47.057+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:47.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:47.058+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:47.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:47.636+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:47.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:47.636+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:47.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:47.636+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:47.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:47.636+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:47.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:47.716+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:47.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:47.716+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:47.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:47.716+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:48.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:47.716+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:48.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:48.011+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:48.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:48.011+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:48.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:48.011+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:48.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:48.012+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:48.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:48.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:48.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:48.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:48.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:48.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:48.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:48.661+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:48.667+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:48.667+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:48.667+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:48.667+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:49.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:48.987+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:49.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:48.987+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:49.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:48.987+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:48.987+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:49.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:49.679+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:49.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:49.679+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:49.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:49.679+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:49.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:49.680+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:49.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:49.615+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:49.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:49.615+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:49.615+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:49.615+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:49.963+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:49.963+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:49.963+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:50.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:49.963+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:50.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:50.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:50.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:50.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:50.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:50.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:50.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:50.612+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:50.925 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:50.726+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:50.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:50.726+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:50.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:50.726+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:50.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:50.727+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:51.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:59:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:59:50.923+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 11966 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:59:51.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:51.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:51.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:51.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:51.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:51.002+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:51.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:51.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:51.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:51.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:51.689+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:51.613+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:51.613+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:51.613+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:51.614+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:52.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:52.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:52.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:52.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:52.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:52.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:52.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:52.000+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:52.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:52.610+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:52.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:52.610+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:52.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:52.610+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:52.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:52.611+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:52.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:52.686+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:52.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:52.686+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:52.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:52.686+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:52.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:52.687+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:53.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:52.953+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:53.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:52.953+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:53.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:52.953+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:52.954+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:53.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:53.628+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:53.628+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:53.628+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:53.629+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:53.696+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:53.696+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:53.696+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:53.697+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:54.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:53.992+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:54.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:53.992+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:54.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:53.992+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:53.993+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:54.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:54.675+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:54.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:54.675+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:54.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:54.675+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:54.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:54.675+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:54.694+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:54.694+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:54.694+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:54.694+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:55.403 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:54.983+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:55.403 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:54.983+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:55.404 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:54.983+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:55.404 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:54.983+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:55.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:55.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:55.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:55.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:55.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:55.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:55.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:55.677+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:55.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:55.664+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:55.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:55.664+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:55.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:55.664+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:55.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:55.664+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:56.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 19:59:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T19:59:55.924+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 12057 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T19:59:56.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:55.940+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:56.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:55.940+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:55.940+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:55.941+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:56.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:56.636+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:56.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:56.636+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:56.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:56.636+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:56.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:56.637+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:56.912 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:56.657+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:56.912 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:56.657+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:56.912 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:56.657+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:56.913 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:56.657+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:56.910+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:56.910+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:56.910+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:56.910+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:57.908 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:57.645+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:57.909 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:57.645+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:57.909 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:57.645+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:57.909 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:57.646+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:57.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:57.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:57.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:57.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:57.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:57.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:57.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:57.644+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:57.905+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:57.905+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:57.905+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:57.906+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:58.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:58.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:58.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:58.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:58.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:58.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:58.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:58.612+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:58.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:58.642+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:58.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:58.642+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:58.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:58.642+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:58.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:58.643+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:59.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:58.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:59.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:58.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:59.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:58.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T19:59:59.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:58.880+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T19:59:59.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:59.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:59.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:59.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:59.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:59.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T19:59:59.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 19:59:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T19:59:59.580+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T19:59:59.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:59.674+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:59.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:59.674+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:59.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:59.674+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T19:59:59.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 19:59:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T19:59:59.674+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:59.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:59.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:59.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:00.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 19:59:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T19:59:59.916+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:00.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:00.606+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:00.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:00.606+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:00.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:00.606+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:00.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:00.607+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:00.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:00.709+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:00.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:00.709+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:00.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:00.709+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:00.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:00.709+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:01.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:00:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:00:00.926+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 12151 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:00:01.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:00.965+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:01.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:00.965+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:01.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:00.965+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:01.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:00.966+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:01.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:01.622+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:01.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:01.622+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:01.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:01.622+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:01.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:01.623+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:01.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:01.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:01.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:01.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:01.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:01.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:01.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:01.667+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:02.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:01.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:02.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:01.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:02.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:01.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:02.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:01.934+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:02.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:02.652+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:02.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:02.652+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:02.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:02.652+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:02.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:02.652+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:02.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:02.698+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:02.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:02.698+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:02.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:02.698+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:02.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:02.699+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:02.924+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:02.924+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:02.924+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:02.925+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:03.951 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:03.658+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:03.951 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:03.658+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:03.951 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:03.658+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:03.951 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:03.658+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:03.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:03.611+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:03.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:03.611+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:03.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:03.611+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:03.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:03.612+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:03.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:03.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:03.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:03.949+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:04.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:04.622+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:04.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:04.622+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:04.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:04.622+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:04.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:04.622+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:04.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:04.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:04.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:04.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:04.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:04.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:04.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:04.689+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:04.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:04.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:04.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:04.934+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:05.930 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:05.627+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:05.930 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:05.627+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:05.930 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:05.627+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:05.930 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:05.628+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:05.930 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:05.690+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:05.930 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:05.690+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:05.930 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:05.690+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:05.930 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:05.691+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:06.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:00:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:00:05.927+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 12242 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:00:06.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:05.932+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:06.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:05.932+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:06.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:05.932+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:05.933+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:06.893 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:06.586+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:06.893 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:06.586+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:06.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:06.586+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:06.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:06.586+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:06.894 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:06.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:06.894 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:06.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:06.894 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:06.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:06.894 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:06.692+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:06.891+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:06.891+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:06.891+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:06.892+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:07.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:07.600+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:07.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:07.600+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:07.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:07.600+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:07.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:07.601+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:07.915 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:07.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:07.915 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:07.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:07.915 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:07.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:07.915 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:07.667+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:08.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:07.912+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:08.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:07.912+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:08.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:07.912+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:07.913+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:08.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:08.636+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:08.919 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:08.636+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:08.919 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:08.636+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:08.919 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:08.637+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:08.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:08.589+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:08.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:08.589+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:08.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:08.589+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:08.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:08.590+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:09.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:08.916+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:09.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:08.916+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:09.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:08.916+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:09.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:08.917+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:09.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:09.674+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:09.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:09.674+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:09.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:09.674+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:09.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:09.675+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:09.574+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:09.574+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:09.574+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:09.574+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:10.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:09.967+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:10.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:09.967+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:10.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:09.967+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:10.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:09.968+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:10.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:10.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:10.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:10.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:10.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:10.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:10.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:10.582+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:10.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:10.698+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:10.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:10.698+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:10.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:10.698+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:10.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:10.699+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:11.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:00:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:00:10.929+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 12338 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:00:11.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:10.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:10.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:10.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:10.989+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:11.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:11.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:11.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:11.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:11.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:11.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:11.564+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:11.675+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:11.675+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:11.675+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:11.676+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:12.433 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:12.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:12.433 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:12.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:12.433 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:12.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:12.433 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:12.002+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:12.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:12.561+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:12.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:12.561+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:12.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:12.561+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:12.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:12.562+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:12.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:12.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:12.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:12.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:12.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:12.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:12.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:12.692+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:12.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:12.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:12.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:12.963+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:13.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:13.539+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:13.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:13.539+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:13.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:13.539+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:13.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:13.540+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:13.943 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:13.724+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:13.943 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:13.724+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:13.943 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:13.724+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:13.943 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:13.724+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:13.940+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:13.940+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:13.940+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:13.941+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:14.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:14.495+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:14.495+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:14.495+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:14.496+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:14.728+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:14.728+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:14.728+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:14.728+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:15.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:14.969+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:15.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:14.969+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:15.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:14.969+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:15.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:14.970+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:15.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:15.527+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:15.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:15.527+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:15.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:15.527+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:15.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:15.527+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:15.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:15.703+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:15.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:15.703+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:15.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:15.703+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:15.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:15.704+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:16.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:00:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:00:15.930+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 12430 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:00:16.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:15.977+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:15.977+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:15.977+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:15.978+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:16.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:16.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:16.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:16.490+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:17.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:16.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:17.219 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:16.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:17.219 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:16.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:17.219 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:16.939+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:17.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:16.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:17.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:16.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:17.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:16.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:17.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:16.749+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:17.681 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:17.445+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:17.682 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:17.445+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:17.682 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:17.445+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:17.682 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:17.446+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:17.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:17.895+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:17.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:17.895+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:17.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:17.895+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:17.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:17.895+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:17.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:17.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:17.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:17.773+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:18.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:18.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:18.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:18.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:18.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:18.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:18.432+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:19.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:18.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:19.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:18.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:19.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:18.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:19.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:18.807+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:18.847+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:18.847+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:18.847+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:18.848+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:19.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:19.447+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:19.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:19.447+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:19.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:19.447+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:19.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:19.448+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:20.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:19.803+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:20.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:19.803+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:20.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:19.803+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:20.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:19.804+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:20.150 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:19.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:20.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:19.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:20.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:19.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:20.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:19.816+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:20.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T20:00:20.335+0000 7f22fc5f0640 0 monclient(hunting): authenticate timed out after 300 2024-09-20T20:00:20.338 INFO:teuthology.orchestra.run.smithi114.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-09-20T20:00:20.697 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:20.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:20.697 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:20.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:20.697 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:20.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:20.697 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:20.437+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:20.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:00:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:00:20.932+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 12524 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:00:20.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:20.813+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:20.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:20.813+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:20.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:20.813+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:20.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:20.813+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:20.775+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:20.775+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:20.775+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:20.776+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:21.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T20:00:21.165 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephadm.py", line 1525, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephadm.py", line 111, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi114 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-09-20T20:00:21.373 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a9bc070680ac4a29b9541a5b9826fc8d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephadm.py", line 1525, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephadm.py", line 111, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi114 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-09-20T20:00:21.376 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-20T20:00:21.387 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-20T20:00:21.387 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T20:00:21.387 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-20T20:00:21.410 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T20:00:21.410 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-20T20:00:21.466 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-20T20:00:21.766 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T20:00:21.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:21.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:21.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:21.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:21.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:21.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:21.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:21.479+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:22.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:21.773+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:22.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:21.773+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:22.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:21.773+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:22.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:21.773+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:22.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:21.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:22.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:21.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:22.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:21.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:22.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:21.824+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:22.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:22.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:22.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:22.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:22.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:22.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:22.432+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:23.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:22.814+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:23.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:22.814+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:23.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:22.814+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:23.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:22.815+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:22.840+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:22.840+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:22.840+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:22.840+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:23.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:23.419+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:23.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:23.419+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:23.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:23.419+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:23.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:23.420+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:24.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:23.846+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:24.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:23.846+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:24.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:23.846+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:24.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:23.847+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:23.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:23.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:23.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:23.796+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:24.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:24.449+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:24.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:24.449+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:24.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:24.449+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:24.450+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:24.826+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:24.826+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:25.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:24.826+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:25.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:24.827+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:24.815+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:24.815+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:24.815+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:24.816+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:25.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:25.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:25.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:25.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:25.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:25.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:25.442+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:26.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:00:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:00:25.933+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 12617 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:00:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:25.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:25.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:25.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:25.865+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:25.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:25.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:25.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:26.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:25.791+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:26.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:26.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:26.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:26.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:26.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:26.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:26.425+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:27.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:26.771+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:26.771+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:26.771+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:26.771+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:26.841+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:27.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:26.841+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:27.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:26.841+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:27.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:26.842+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:27.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:27.465+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:27.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:27.465+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:27.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:27.465+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:27.767 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:27.465+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:28.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:27.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:28.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:27.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:27.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:27.764+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:27.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:27.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:27.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:27.851+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:28.837 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:28.474+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:28.837 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:28.474+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:28.837 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:28.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:28.837 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:28.475+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:28.838 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:28.717+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:28.838 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:28.717+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:28.838 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:28.717+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:28.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:28.718+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:29.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:28.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:29.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:28.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:29.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:28.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:28.834+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:29.843 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:29.674+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:29.843 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:29.674+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:29.843 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:29.674+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:29.843 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:29.675+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:29.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:29.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:29.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:29.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:29.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:29.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:29.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:29.487+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:30.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:29.840+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:30.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:29.840+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:30.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:29.840+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:30.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:29.841+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:30.797 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:30.628+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:30.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:30.628+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:30.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:30.628+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:30.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:30.629+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:30.798 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:30.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:30.798 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:30.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:30.798 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:30.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:30.799 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:30.525+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:31.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:00:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:00:30.934+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 12712 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:00:31.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:30.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:31.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:30.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:30.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:30.795+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:31.808 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:31.671+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:31.809 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:31.671+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:31.809 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:31.671+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:31.809 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:31.672+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:31.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:31.537+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:31.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:31.537+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:31.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:31.537+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:31.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:31.538+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:32.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:31.805+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:32.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:31.805+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:32.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:31.805+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:31.806+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:32.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:32.681+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:32.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:32.681+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:32.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:32.681+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:32.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:32.681+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:32.840 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:32.504+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:32.841 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:32.504+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:32.841 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:32.504+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:32.841 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:32.504+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:33.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:32.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:33.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:32.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:32.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:32.837+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:33.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:33.699+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:33.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:33.699+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:33.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:33.699+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:33.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:33.700+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:33.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:33.458+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:33.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:33.458+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:33.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:33.458+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:33.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:33.458+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:34.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:33.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:34.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:33.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:33.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:33.792+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:34.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:34.674+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:34.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:34.674+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:34.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:34.674+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:34.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:34.674+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:34.428+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:34.428+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:34.428+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:34.429+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:35.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:34.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:35.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:34.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:35.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:34.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:34.748+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:35.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:35.652+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:35.652+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:35.652+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:35.652+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:35.428+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:35.428+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:35.428+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:35.429+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:36.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:00:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:00:35.936+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 12803 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:00:36.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:35.749+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:36.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:35.749+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:36.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:35.749+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:36.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:35.750+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:36.665 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:36.408+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:36.665 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:36.408+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:36.666 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:36.408+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:36.666 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:36.408+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:36.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:36.750+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:36.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:36.750+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:36.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:36.750+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:36.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:36.751+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:36.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:36.663+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:36.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:36.663+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:36.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:36.663+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:36.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:36.664+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:37.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:37.411+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:37.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:37.411+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:37.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:37.411+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:37.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:37.412+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:37.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:37.686+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:37.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:37.686+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:37.686+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:37.687+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:37.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:37.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:37.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:37.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:37.752+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:38.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:38.407+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:38.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:38.407+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:38.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:38.407+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:38.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:38.407+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:38.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:38.644+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:38.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:38.644+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:38.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:38.644+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:38.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:38.645+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:38.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:38.712+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:38.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:38.712+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:38.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:38.712+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:38.713+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:39.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:39.606+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:39.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:39.606+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:39.606+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:39.607+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:39.381+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:39.381+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:39.381+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:39.382+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:40.080 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:39.718+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:40.081 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:39.718+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:40.081 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:39.718+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:40.081 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:39.719+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:40.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:40.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:40.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:40.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:40.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:40.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:40.424+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:40.562+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:40.562+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:40.562+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:40.563+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:41.115 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:00:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:00:40.938+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 12897 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:00:41.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:40.757+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:41.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:40.757+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:41.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:40.757+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:41.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:40.758+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:41.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:41.471+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:41.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:41.471+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:41.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:41.471+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:41.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:41.472+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:41.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:41.552+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:41.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:41.552+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:41.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:41.552+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:41.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:41.553+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:41.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:41.711+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:41.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:41.711+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:41.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:41.711+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:41.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:41.712+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:42.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:42.668+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:42.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:42.668+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:42.668+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:42.668+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:42.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:42.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:42.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:42.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:42.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:42.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:42.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:42.467+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:42.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:42.523+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:42.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:42.523+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:42.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:42.523+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:42.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:42.524+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:43.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:43.488+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:43.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:43.488+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:43.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:43.488+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:43.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:43.489+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:43.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:43.450+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:43.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:43.450+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:43.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:43.450+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:43.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:43.451+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:43.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:43.703+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:43.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:43.703+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:43.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:43.704+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:43.704+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:44.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:44.684+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:44.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:44.684+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:44.684+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:44.685+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:44.463+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:44.463+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:44.463+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:44.463+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:44.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:44.461+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:44.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:44.461+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:44.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:44.461+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:44.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:44.462+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:45.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:45.512+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:45.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:45.512+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:45.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:45.512+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:45.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:45.512+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:45.943 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:45.655+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:45.943 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:45.655+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:45.943 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:45.655+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:45.943 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:45.655+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:45.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:45.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:45.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:45.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:45.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:45.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:45.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:45.507+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:00:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:00:45.940+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 12990 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:00:46.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:46.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:46.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:46.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:46.531+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:46.643+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:46.643+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:46.643+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:46.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:46.643+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:46.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:46.491+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:46.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:46.491+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:46.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:46.491+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:46.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:46.493+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:47.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:47.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:47.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:47.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:47.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:47.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:47.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:47.569+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:47.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:47.491+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:47.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:47.491+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:47.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:47.491+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:47.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:47.492+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:47.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:47.686+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:47.686+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:47.686+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:47.687+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:48.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:48.469+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:48.469+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:48.469+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:48.469+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:48.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:48.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:48.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:48.573+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:48.649+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:48.649+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:48.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:48.649+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:48.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:48.649+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:49.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:49.466+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:49.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:49.466+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:49.466+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:49.467+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:49.618+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:49.618+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:49.618+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:49.619+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:49.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:49.696+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:49.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:49.696+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:49.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:49.696+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:49.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:49.697+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:50.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:50.487+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:50.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:50.487+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:50.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:50.487+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:50.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:50.488+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:50.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:50.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:50.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:50.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:50.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:50.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:50.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:50.640+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:50.699+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:50.699+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:50.699+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:50.699+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:51.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:00:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:00:50.941+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 13085 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:00:51.609 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:51.482+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:51.609 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:51.482+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:51.609 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:51.482+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:51.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:51.483+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:51.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:51.607+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:51.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:51.607+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:51.607+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:51.608+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:51.677+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:51.677+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:51.677+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:51.678+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:52.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:52.475+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:52.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:52.475+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:52.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:52.475+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:52.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:52.476+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:52.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:52.695+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:52.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:52.695+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:52.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:52.695+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:52.696+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:52.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:52.628+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:52.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:52.628+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:52.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:52.628+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:52.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:52.629+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:53.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:53.477+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:53.477+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:53.477+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:53.478+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:53.677+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:53.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:53.677+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:53.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:53.677+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:53.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:53.677+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:53.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:53.662+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:53.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:53.662+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:53.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:53.662+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:53.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:53.663+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:54.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:54.680+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:54.680+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:54.680+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:54.680+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:54.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:54.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:54.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:54.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:54.474+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:54.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:54.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:54.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:54.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:54.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:54.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:54.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:54.712+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:55.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:55.455+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:55.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:55.455+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:55.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:55.455+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:55.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:55.456+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:55.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:55.665+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:55.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:55.665+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:55.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:55.665+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:55.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:55.665+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:55.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:00:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:00:55.943+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 13176 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:00:55.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:55.712+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:55.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:55.712+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:55.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:55.712+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:55.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:55.712+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:56.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:56.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:56.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:56.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:56.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:56.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:56.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:56.436+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:56.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:56.762+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:56.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:56.762+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:56.762+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:56.763+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:56.677+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:56.677+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:56.677+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:56.677+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:57.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:57.442+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:57.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:57.442+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:57.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:57.442+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:57.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:57.442+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:57.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:57.786+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:57.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:57.786+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:57.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:57.786+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:57.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:57.787+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:57.698+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:57.698+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:57.698+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:57.699+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:58.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:58.422+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:58.422+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:58.422+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:58.423+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:00:59.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:58.718+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:59.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:58.718+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:58.718+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:00:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:58.719+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:58.740+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:58.740+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:58.740+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:00:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:58.741+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:00:59.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:59.454+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:59.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:59.454+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:59.454+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:00:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:00:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:00:59.455+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:00.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:59.753+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:00.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:59.753+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:00.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:59.753+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:00.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:00:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:00:59.754+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:00.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:59.732+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:00.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:59.732+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:00.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:59.732+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:00.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:00:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:00:59.733+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:00.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:00.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:00.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:00.413+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:01.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:01:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:01:00.944+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 13270 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:01:01.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:00.739+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:01.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:00.739+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:01.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:00.739+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:01.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:00.739+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:01.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:00.750+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:01.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:00.750+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:01.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:00.750+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:01.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:00.750+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:01.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:01.434+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:01.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:01.434+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:01.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:01.434+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:01.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:01.435+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:01.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:01.698+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:01.698+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:01.698+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:01.699+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:01.747+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:01.747+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:01.747+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:01.748+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:02.708 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:02.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:02.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:02.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:02.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:02.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:02.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:02.441+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:02.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:02.706+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:02.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:02.706+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:02.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:02.706+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:02.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:02.707+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:02.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:02.724+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:02.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:02.724+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:02.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:02.724+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:02.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:02.724+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:03.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:03.394+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:03.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:03.394+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:03.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:03.394+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:03.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:03.395+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:03.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:03.707+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:03.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:03.707+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:03.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:03.707+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:03.708+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:03.725+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:03.725+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:03.725+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:03.726+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:04.681 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:04.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:04.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:04.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:04.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:04.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:04.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:04.413+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:04.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:04.679+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:04.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:04.679+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:04.679+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:04.680+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:04.702+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:04.702+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:04.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:04.702+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:04.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:04.703+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:05.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:05.449+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:05.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:05.449+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:05.449+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:05.450+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:05.655+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:05.655+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:05.655+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:05.656+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:05.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:05.695+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:05.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:05.695+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:05.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:05.695+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:05.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:05.696+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:06.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:01:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:01:05.946+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 13362 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:01:06.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:06.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:06.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:06.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:06.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:06.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:06.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:06.401+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:06.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:06.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:06.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:06.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:06.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:06.691+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:06.692+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:06.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:06.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:06.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:06.693+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:07.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:07.427+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:07.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:07.427+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:07.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:07.427+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:07.428+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:07.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:07.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:07.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:07.645+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:08.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:07.721+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:08.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:07.721+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:07.721+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:07.722+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:08.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:08.619+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:08.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:08.619+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:08.619+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:08.620+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:08.454+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:08.454+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:08.454+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:08.455+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:09.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:08.754+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:09.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:08.754+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:09.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:08.754+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:08.755+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:09.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:09.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:09.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:09.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:09.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:09.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:09.621+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:09.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:09.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:09.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:09.440+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:10.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:09.730+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:10.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:09.730+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:10.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:09.730+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:10.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:09.730+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:10.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:10.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:10.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:10.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:10.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:10.664+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:10.445+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:10.445+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:10.445+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:10.445+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:11.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:01:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:01:10.947+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 13457 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:01:11.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:10.761+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:10.761+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:10.761+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:10.762+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:11.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:11.444+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:11.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:11.444+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:11.615 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:11.444+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:11.615 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:11.445+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:11.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:11.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:11.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:11.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:11.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:11.665+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:11.782+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:11.782+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:11.782+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:11.782+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:12.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:12.655+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:12.655+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:12.655+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:12.655+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:12.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:12.407+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:12.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:12.407+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:12.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:12.407+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:12.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:12.408+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:13.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:12.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:13.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:12.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:12.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:12.817+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:13.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:13.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:13.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:13.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:13.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:13.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:13.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:13.414+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:13.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:13.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:13.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:13.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:13.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:13.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:13.832+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:13.684+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:13.684+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:13.684+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:13.685+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:14.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:14.452+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:14.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:14.452+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:14.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:14.452+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:14.452+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:14.653+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:14.653+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:14.653+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:14.654+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:14.864+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:14.864+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:14.864+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:14.865+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:15.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:15.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:15.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:15.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:15.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:15.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:15.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:15.413+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:15.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:15.689+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:15.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:15.689+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:15.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:15.689+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:15.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:15.689+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:15.952 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:15.884+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:15.952 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:15.884+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:15.952 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:15.884+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:15.952 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:15.885+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:16.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:01:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:01:15.948+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 13548 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:01:16.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:16.459+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:16.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:16.459+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:16.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:16.459+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:16.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:16.459+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:16.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:16.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:16.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:16.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:16.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:16.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:16.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:16.672+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:17.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:16.925+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:17.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:16.925+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:17.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:16.925+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:17.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:16.926+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:17.474 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:17.436+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:17.474 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:17.436+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:17.474 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:17.436+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:17.474 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:17.437+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:17.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:17.691+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:17.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:17.691+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:17.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:17.691+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:17.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:17.691+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:18.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:17.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:18.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:17.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:17.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:17.881+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:18.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:18.449+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:18.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:18.449+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:18.449+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:18.450+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:18.643+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:18.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:18.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:18.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:18.644+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:19.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:18.892+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:19.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:18.892+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:19.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:18.892+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:18.893+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:19.857 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:19.494+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:19.857 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:19.494+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:19.857 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:19.494+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:19.857 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:19.494+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:19.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:19.622+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:19.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:19.622+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:19.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:19.622+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:19.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:19.623+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:20.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:19.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:20.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:19.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:20.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:19.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:20.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:19.856+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:20.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:20.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:20.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:20.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:20.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:20.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:20.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:20.472+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:20.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:20.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:20.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:20.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:20.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:20.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:20.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:20.580+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:21.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:01:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:01:20.950+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 13643 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:01:21.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:20.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:21.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:20.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:20.809+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:20.809+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:21.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:21.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:21.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:21.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:21.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:21.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:21.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:21.474+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:21.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:21.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:21.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:21.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:21.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:21.598+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:21.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:21.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:21.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:21.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:21.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:21.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:21.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:21.832+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:22.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:22.452+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:22.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:22.452+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:22.452+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:22.453+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:22.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:22.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:22.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:22.579+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:23.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:22.827+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:23.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:22.827+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:23.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:22.827+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:23.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:22.828+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:23.818 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:23.476+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:23.818 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:23.476+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:23.819 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:23.476+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:23.819 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:23.476+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:23.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:23.557+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:23.820 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:23.557+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:23.820 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:23.557+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:23.820 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:23.558+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:24.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:23.813+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:24.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:23.813+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:24.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:23.813+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:23.814+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:24.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:24.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:24.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:24.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:24.436+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:24.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:24.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:24.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:24.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:24.516+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:25.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:24.796+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:25.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:24.796+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:24.796+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:24.796+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:25.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:25.422+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:25.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:25.423+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:25.423+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:25.423+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:25.495+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:25.495+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:25.495+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:25.496+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:26.161 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:01:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:01:25.952+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 13735 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:01:26.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:25.829+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:26.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:25.829+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:26.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:25.829+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:26.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:25.829+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:26.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:26.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:26.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:26.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:26.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:26.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:26.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:26.408+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:26.823 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:26.483+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:26.823 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:26.483+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:26.823 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:26.483+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:26.823 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:26.484+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:27.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:26.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:26.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:26.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:26.821+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:27.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:27.376+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:27.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:27.376+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:27.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:27.376+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:27.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:27.377+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:27.498+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:27.498+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:27.498+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:27.499+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:28.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:27.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:28.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:27.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:28.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:27.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:27.843+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:28.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:28.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:28.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:28.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:28.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:28.331+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:28.544+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:28.544+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:28.544+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:28.545+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:29.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:28.854+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:28.854+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:28.854+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:28.855+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:29.563 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:29.296+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:29.563 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:29.296+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:29.563 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:29.296+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:29.564 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:29.296+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:29.840 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:29.561+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:29.840 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:29.561+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:29.840 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:29.561+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:29.840 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:29.562+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:30.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:29.838+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:30.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:29.838+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:30.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:29.838+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:30.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:29.839+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:30.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:30.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:30.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:30.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:30.346+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:30.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:30.591+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:30.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:30.591+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:30.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:30.591+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:30.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:30.592+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:30.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:30.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:30.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:30.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:30.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:30.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:30.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:30.835+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:31.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:01:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:01:30.954+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 13831 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:01:31.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:31.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:31.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:31.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:31.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:31.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:31.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:31.331+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:31.911 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:31.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:31.911 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:31.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:31.911 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:31.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:31.911 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:31.812+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:31.911 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:31.594+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:31.911 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:31.594+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:31.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:31.594+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:31.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:31.595+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:32.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:32.290+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:32.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:32.290+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:32.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:32.290+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:32.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:32.290+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:32.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:32.631+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:32.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:32.631+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:32.631+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:32.632+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:32.776+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:32.776+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:32.776+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:32.777+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:33.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:33.267+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:33.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:33.267+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:33.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:33.267+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:33.573 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:33.268+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:33.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:33.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:33.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:33.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:33.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:33.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:33.672+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:33.770+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:33.770+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:33.770+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:33.770+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:34.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:34.266+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:34.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:34.266+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:34.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:34.266+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:34.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:34.267+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:34.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:34.741+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:34.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:34.741+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:34.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:34.741+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:34.742+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:34.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:34.709+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:34.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:34.709+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:34.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:34.709+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:34.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:34.709+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:35.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:35.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:35.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:35.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:35.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:35.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:35.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:35.280+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:35.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:35.677+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:35.958 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:35.677+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:35.958 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:35.677+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:35.958 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:35.678+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:35.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:35.733+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:35.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:35.733+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:35.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:35.733+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:35.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:35.734+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:36.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:01:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:01:35.955+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 13921 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:01:36.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:36.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:36.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:36.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:36.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:36.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:36.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:36.277+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:36.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:36.656+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:36.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:36.656+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:36.656+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:36.657+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:36.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:36.740+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:36.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:36.740+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:36.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:36.740+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:36.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:36.740+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:37.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:37.325+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:37.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:37.325+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:37.656 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:37.325+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:37.656 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:37.326+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:37.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:37.653+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:37.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:37.653+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:37.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:37.653+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:37.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:37.654+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:37.715+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:37.715+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:37.715+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:37.716+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:38.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:38.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:38.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:38.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:38.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:38.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:38.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:38.333+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:38.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:38.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:38.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:38.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:38.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:38.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:38.640+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:38.744+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:38.744+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:38.744+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:38.745+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:39.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:39.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:39.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:39.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:39.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:39.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:39.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:39.331+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:39.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:39.621+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:39.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:39.621+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:39.621+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:39.621+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:39.776+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:39.777+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:39.777+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:39.777+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:40.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:40.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:40.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:40.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:40.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:40.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:40.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:40.299+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:40.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:40.669+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:40.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:40.669+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:40.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:40.669+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:40.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:40.670+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:40.960 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:40.779+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:40.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:40.779+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:40.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:40.779+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:40.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:40.780+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:01:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:01:40.957+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 14015 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:01:41.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:41.296+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:41.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:41.296+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:41.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:41.296+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:41.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:41.296+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:41.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:41.706+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:41.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:41.706+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:41.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:41.706+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:41.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:41.706+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:41.774 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:41.771+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:41.774 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:41.771+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:41.774 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:41.771+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:42.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:41.771+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:42.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:42.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:42.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:42.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:42.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:42.276+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:42.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:42.697+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:42.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:42.697+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:42.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:42.697+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:42.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:42.698+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:42.794+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:42.794+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:42.794+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:42.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:42.795+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:43.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:43.320+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:43.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:43.320+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:43.573 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:43.320+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:43.573 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:43.320+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:43.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:43.652+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:43.652+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:43.652+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:43.653+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:43.746+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:43.746+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:43.746+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:43.746+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:44.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:44.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:44.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:44.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:44.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:44.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:44.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:44.351+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:44.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:44.690+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:44.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:44.690+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:44.690+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:44.690+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:44.774+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:44.774+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:44.774+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:44.775+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:45.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:45.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:45.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:45.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:45.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:45.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:45.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:45.389+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:45.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:45.707+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:45.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:45.707+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:45.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:45.707+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:45.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:45.708+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:45.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:45.784+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:45.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:45.784+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:45.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:45.784+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:45.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:45.785+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:01:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:01:45.958+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 14106 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:01:46.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:46.393+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:46.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:46.393+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:46.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:46.393+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:46.394+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:47.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:46.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:47.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:46.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:47.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:46.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:47.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:46.723+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:47.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:46.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:47.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:46.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:47.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:46.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:47.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:46.796+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:47.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:47.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:47.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:47.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:47.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:47.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:47.409+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:48.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:47.763+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:48.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:47.763+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:47.763+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:47.764+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:47.785+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:47.785+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:47.785+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:47.786+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:48.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:48.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:48.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:48.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:48.440+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:48.743+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:48.743+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:48.743+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:48.744+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:48.832+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:48.832+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:48.832+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:48.832+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:49.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:49.480+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:49.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:49.480+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:49.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:49.480+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:49.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:49.480+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:50.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:49.736+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:50.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:49.736+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:50.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:49.736+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:50.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:49.737+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:50.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:49.789+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:50.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:49.789+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:50.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:49.789+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:50.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:49.790+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:50.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:50.463+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:50.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:50.463+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:50.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:50.463+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:50.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:50.464+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:51.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:01:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:01:50.960+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 14195 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:01:51.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:50.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:51.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:50.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:51.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:50.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:51.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:50.723+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:51.241 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:50.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:51.241 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:50.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:51.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:50.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:51.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:50.836+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:51.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:51.463+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:51.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:51.463+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:51.463+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:51.464+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:52.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:51.748+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:52.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:51.748+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:51.748+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:51.749+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:51.804+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:51.804+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:51.804+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:51.804+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:52.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:52.479+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:52.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:52.479+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:52.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:52.479+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:52.480+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:52.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:52.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:52.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:52.768+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:52.731+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:52.731+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:52.731+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:52.732+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:53.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:53.478+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:53.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:53.478+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:53.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:53.478+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:53.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:53.479+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:53.794+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:53.794+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:53.794+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:53.795+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:53.754+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:53.754+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:53.754+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:53.755+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:54.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:54.438+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:54.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:54.438+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:54.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:54.438+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:54.438+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:55.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:54.799+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:55.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:54.799+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:54.799+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:54.799+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:54.770+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:54.770+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:54.770+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:54.771+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:55.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:55.423+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:55.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:55.423+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:55.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:55.423+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:55.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:55.423+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:56.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:01:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:01:55.962+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 14281 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:01:56.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:55.778+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:55.778+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:55.778+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:55.779+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:55.778+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:55.778+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:55.778+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:55.779+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:56.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:56.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:56.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:56.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:56.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:56.383+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:57.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:56.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:57.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:56.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:56.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:56.812+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:56.791+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:56.791+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:56.791+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:56.791+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:57.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:57.430+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:57.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:57.430+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:57.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:57.430+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:57.431+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:58.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:57.841+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:58.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:57.841+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:58.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:57.841+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:57.841+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:57.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:57.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:57.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:57.857+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:58.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:58.432+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:58.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:58.432+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:58.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:58.432+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:58.433+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:01:59.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:58.889+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:58.889+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:58.889+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:01:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:58.889+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:58.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:58.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:58.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:01:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:58.887+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:01:59.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:59.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:59.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:59.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:01:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:01:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:01:59.402+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:00.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:59.931+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:00.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:59.931+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:00.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:59.931+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:00.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:01:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:01:59.932+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:00.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:59.884+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:00.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:59.884+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:00.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:59.884+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:00.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:01:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:01:59.884+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:00.415+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:00.415+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:00.415+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:00.415+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:01.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:02:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:02:00.963+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 14370 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:02:01.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:00.920+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:01.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:00.920+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:01.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:00.920+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:01.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:00.921+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:00.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:00.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:00.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:01.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:00.872+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:01.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:01.448+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:01.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:01.448+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:01.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:01.448+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:01.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:01.449+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:02.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:01.931+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:02.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:01.931+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:01.931+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:01.932+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:01.825+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:01.825+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:01.825+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:01.826+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:02.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:02.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:02.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:02.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:02.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:02.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:02.471+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:03.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:02.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:03.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:02.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:03.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:02.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:02.891+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:02.860+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:02.860+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:02.860+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:02.861+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:03.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:03.450+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:03.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:03.450+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:03.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:03.450+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:03.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:03.450+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:03.827+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:03.827+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:03.827+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:03.827+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:03.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:03.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:03.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:03.934+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:04.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:04.432+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:04.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:04.432+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:04.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:04.432+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:04.433+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:05.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:04.802+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:05.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:04.802+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:04.802+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:04.803+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:04.936+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:04.936+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:04.936+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:04.937+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:05.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:05.445+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:05.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:05.445+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:05.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:05.445+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:05.446+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:06.096 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:02:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:02:05.966+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 14456 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:02:06.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:05.909+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:06.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:05.909+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:06.097 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:05.909+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:06.097 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:05.909+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:06.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:05.784+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:06.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:05.784+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:06.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:05.784+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:06.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:05.785+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:06.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:06.421+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:06.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:06.421+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:06.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:06.421+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:06.422+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:07.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:06.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:07.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:06.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:07.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:06.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:07.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:06.917+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:06.774+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:06.774+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:06.774+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:06.775+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:07.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:07.372+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:07.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:07.372+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:07.372+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:07.373+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:08.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:07.941+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:08.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:07.941+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:08.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:07.941+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:07.941+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:07.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:07.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:07.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:07.749+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:08.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:08.420+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:08.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:08.420+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:08.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:08.420+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:08.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:08.421+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:08.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:08.710+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:08.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:08.710+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:08.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:08.710+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:08.710+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:09.418 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:08.977+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:09.418 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:08.977+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:09.418 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:08.977+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:09.418 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:08.978+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:09.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:09.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:09.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:09.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:09.690 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:09.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:09.690 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:09.416+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:09.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:09.687+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:09.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:09.687+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:09.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:09.687+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:09.688+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:10.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:09.972+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:10.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:09.972+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:10.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:09.972+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:10.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:09.973+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:10.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:10.375+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:10.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:10.375+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:10.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:10.375+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:10.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:10.375+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:10.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:10.941+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:10.941+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:10.941+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:10.942+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:10.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:10.708+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:10.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:10.708+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:10.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:10.708+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:10.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:10.709+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:11.262 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:02:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:02:10.967+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 14546 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:02:11.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:11.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:11.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:11.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:11.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:11.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:11.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:11.413+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:11.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:11.680+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:11.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:11.680+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:11.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:11.680+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:11.681+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:12.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:11.963+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:12.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:11.963+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:12.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:11.963+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:12.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:11.964+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:12.558 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:12.457+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:12.558 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:12.457+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:12.558 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:12.457+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:12.559 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:12.458+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:12.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:12.662+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:12.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:12.662+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:12.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:12.662+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:12.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:12.663+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:13.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:12.980+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:13.449 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:12.980+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:13.449 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:12.980+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:13.449 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:12.981+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:13.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:13.446+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:13.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:13.446+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:13.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:13.446+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:13.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:13.447+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:13.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:13.683+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:13.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:13.683+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:13.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:13.683+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:13.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:13.684+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:13.950+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:13.950+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:13.950+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:13.951+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:14.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:14.451+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:14.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:14.451+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:14.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:14.451+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:14.451+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:14.672+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:14.672+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:14.672+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:14.672+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:15.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:14.960+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:15.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:14.960+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:15.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:14.960+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:15.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:14.961+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:15.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:15.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:15.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:15.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:15.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:15.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:15.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:15.474+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:15.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:15.625+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:15.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:15.625+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:15.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:15.625+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:15.626+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:16.374 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:02:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:02:15.969+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 14632 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:02:16.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:16.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:16.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:16.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:16.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:16.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:16.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:16.003+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:16.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:16.517+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:16.517+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:16.517+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:16.518+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:16.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:16.608+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:16.608+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:16.608+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:16.608+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:17.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:17.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:17.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:17.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:17.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:17.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:17.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:17.003+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:17.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:17.560+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:17.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:17.560+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:17.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:17.560+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:17.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:17.561+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:17.597+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:17.597+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:17.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:17.597+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:17.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:17.598+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:18.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:17.997+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:18.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:17.997+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:18.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:17.997+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:18.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:17.998+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:18.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:18.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:18.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:18.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:18.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:18.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:18.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:18.593+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:18.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:18.593+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:18.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:18.593+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:18.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:18.593+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:18.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:18.594+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:19.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:18.955+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:19.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:18.955+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:19.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:18.955+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:18.956+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:19.934 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:19.609+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:19.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:19.609+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:19.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:19.609+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:19.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:19.610+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:19.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:19.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:19.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:19.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:19.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:19.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:19.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:19.572+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:20.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:19.932+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:20.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:19.932+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:20.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:19.932+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:20.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:19.933+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:20.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:20.618+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:20.933 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:20.618+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:20.933 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:20.618+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:20.933 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:20.619+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:20.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:20.547+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:20.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:20.547+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:20.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:20.547+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:20.934 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:20.548+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:21.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:02:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:02:20.971+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 14721 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:02:21.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:20.930+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:21.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:20.930+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:21.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:20.930+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:21.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:20.931+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:21.960 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:21.531+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:21.960 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:21.531+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:21.960 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:21.531+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:21.960 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:21.532+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:21.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:21.634+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:21.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:21.634+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:21.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:21.634+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:21.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:21.635+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:22.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:21.958+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:22.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:21.958+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:22.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:21.958+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:22.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:21.958+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:22.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:22.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:22.919 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:22.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:22.919 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:22.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:22.919 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:22.575+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:22.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:22.661+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:22.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:22.661+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:22.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:22.661+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:22.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:22.662+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:23.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:22.916+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:23.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:22.916+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:23.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:22.916+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:23.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:22.917+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:23.872 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:23.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:23.872 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:23.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:23.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:23.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:23.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:23.668+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:23.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:23.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:23.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:23.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:23.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:23.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:23.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:23.592+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:24.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:23.870+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:24.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:23.870+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:24.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:23.870+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:24.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:23.871+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:24.878 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:24.632+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:24.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:24.632+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:24.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:24.632+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:24.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:24.632+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:24.879 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:24.611+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:24.879 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:24.611+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:24.879 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:24.611+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:24.879 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:24.612+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:25.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:24.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:25.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:24.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:25.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:24.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:25.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:24.877+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:25.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:25.657+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:25.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:25.657+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:25.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:25.657+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:25.658+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:25.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:25.888+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:25.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:25.888+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:25.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:25.888+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:25.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:25.888+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:25.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:25.653+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:25.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:25.653+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:25.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:25.653+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:25.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:25.654+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:26.368 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:02:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:02:25.972+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 14806 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:02:26.680 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:26.613+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:26.681 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:26.613+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:26.681 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:26.613+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:26.681 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:26.613+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:26.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:26.679+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:26.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:26.679+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:26.679+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:26.679+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:26.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:26.908+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:26.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:26.908+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:26.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:26.908+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:26.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:26.909+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:27.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:27.677+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:27.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:27.677+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:27.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:27.677+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:27.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:27.678+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:27.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:27.649+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:27.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:27.649+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:27.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:27.649+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:27.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:27.650+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:28.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:27.955+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:28.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:27.955+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:28.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:27.955+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:27.955+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:28.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:28.693+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:28.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:28.693+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:28.693+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:28.694+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:28.679+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:28.679+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:28.679+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:28.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:28.680+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:28.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:28.921+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:28.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:28.921+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:28.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:28.921+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:28.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:28.922+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:29.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:29.685+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:29.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:29.685+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:29.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:29.685+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:29.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:29.686+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:29.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:29.700+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:29.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:29.700+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:29.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:29.700+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:29.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:29.700+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:30.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:29.960+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:30.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:29.960+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:29.960+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:29.961+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:30.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:30.677+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:30.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:30.677+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:30.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:30.677+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:30.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:30.678+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:30.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:30.714+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:30.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:30.714+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:30.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:30.714+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:30.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:30.715+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:31.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:02:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:02:30.974+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 14895 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:02:31.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:30.953+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:31.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:30.953+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:31.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:30.953+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:31.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:30.954+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:31.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:31.753+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:31.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:31.753+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:31.753+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:31.754+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:31.700+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:31.700+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:31.700+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:31.701+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:32.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:31.971+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:32.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:31.971+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:32.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:31.971+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:31.972+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:32.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:32.728+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:32.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:32.728+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:32.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:32.728+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:32.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:32.729+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:32.996 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:32.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:32.997 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:32.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:32.997 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:32.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:32.997 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:32.752+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:33.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:32.994+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:33.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:32.994+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:33.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:32.994+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:32.994+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:33.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:33.723+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:33.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:33.723+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:33.723+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:33.724+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:33.801+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:33.801+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:33.801+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:33.802+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:34.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:34.024+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:34.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:34.024+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:34.024+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:34.025+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:35.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:34.770+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:35.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:34.770+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:35.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:34.770+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:34.770+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:34.796+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:34.796+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:34.796+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:34.797+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:35.018+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:35.018+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:35.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:35.018+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:35.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:35.018+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:36.015 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:02:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:02:35.976+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 14980 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:02:36.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:35.815+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:36.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:35.815+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:36.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:35.815+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:36.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:35.816+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:36.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:35.756+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:36.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:35.756+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:36.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:35.756+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:36.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:35.756+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:36.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:36.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:36.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:36.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:36.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:36.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:36.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:36.014+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:36.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:36.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:36.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:36.773+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:36.985+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:36.985+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:36.985+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:36.985+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:37.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:36.744+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:37.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:36.744+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:37.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:36.744+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:37.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:36.745+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:38.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:37.758+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:38.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:37.758+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:38.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:37.758+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:37.758+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:37.787+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:37.787+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:37.787+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:37.788+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:38.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:37.975+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:38.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:37.975+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:38.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:37.975+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:38.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:37.976+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:38.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:39.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:38.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:38.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:38.782+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:38.804+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:38.804+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:38.804+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:38.805+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:39.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:38.960+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:39.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:38.960+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:39.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:38.960+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:39.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:38.960+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:40.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:39.961+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:40.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:39.962+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:40.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:39.962+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:40.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:39.962+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:40.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:39.788+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:40.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:39.788+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:40.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:39.788+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:40.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:39.788+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:40.163 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:39.768+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:40.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:39.768+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:40.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:39.768+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:40.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:39.769+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:40.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:40.775+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:40.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:40.775+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:40.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:40.775+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:40.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:40.775+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:40.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:40.958+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:40.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:40.958+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:40.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:40.958+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:40.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:40.959+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:40.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:40.726+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:40.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:40.726+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:40.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:40.726+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:40.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:40.726+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:41.255 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:02:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:02:40.977+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 15068 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:02:42.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:41.984+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:42.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:41.984+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:42.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:41.984+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:42.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:41.985+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:41.743+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:42.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:41.743+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:42.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:41.743+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:42.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:41.744+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:42.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:41.781+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:42.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:41.781+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:42.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:41.781+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:42.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:41.782+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:42.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:42.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:42.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:42.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:42.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:42.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:42.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:42.783+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:42.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:42.707+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:42.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:42.707+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:42.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:42.707+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:42.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:42.708+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:43.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:42.996+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:43.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:42.996+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:43.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:42.996+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:43.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:42.997+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:43.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:43.765+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:43.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:43.765+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:43.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:43.765+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:43.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:43.765+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:43.959 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:43.667+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:43.959 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:43.667+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:43.959 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:43.667+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:43.959 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:43.667+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:44.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:43.956+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:44.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:43.956+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:43.956+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:43.957+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:44.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:44.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:44.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:44.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:44.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:44.917+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:44.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:44.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:44.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:44.799+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:44.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:44.670+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:44.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:44.670+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:44.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:44.670+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:44.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:44.671+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:45.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:45.919+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:45.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:45.919+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:45.919+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:45.919+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:45.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:45.783+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:45.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:45.783+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:45.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:45.783+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:45.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:45.783+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:45.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:45.710+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:45.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:45.710+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:45.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:45.710+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:45.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:45.711+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:46.363 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:02:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:02:45.979+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 15153 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:02:47.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:46.908+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:46.908+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:46.908+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:46.909+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:46.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:46.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:47.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:46.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:47.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:46.818+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:47.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:46.741+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:47.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:46.741+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:47.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:46.741+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:47.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:46.741+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:48.108 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:47.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:48.108 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:47.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:48.108 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:47.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:48.109 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:47.845+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:48.109 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:47.909+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:48.109 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:47.909+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:48.109 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:47.909+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:48.109 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:47.910+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:48.110 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:47.726+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:48.110 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:47.726+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:48.110 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:47.726+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:48.110 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:47.726+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:48.892+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:48.892+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:48.892+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:48.893+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:48.867+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:48.867+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:48.867+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:48.867+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:48.738+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:48.738+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:48.738+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:48.739+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:50.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:49.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:50.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:49.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:50.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:49.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:50.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:49.878+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:50.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:49.835+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:50.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:49.835+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:50.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:49.835+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:50.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:49.835+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:50.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:49.763+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:50.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:49.763+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:50.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:49.763+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:50.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:49.763+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:51.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:02:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:02:50.980+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 15242 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:02:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:50.744+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:50.744+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:50.744+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:50.744+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:50.864+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:50.864+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:51.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:50.864+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:51.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:50.865+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:51.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:50.904+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:51.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:50.904+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:51.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:50.904+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:51.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:50.906+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:52.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:51.825+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:52.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:51.825+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:52.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:51.825+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:52.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:51.826+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:52.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:51.868+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:52.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:51.868+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:52.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:51.868+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:52.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:51.868+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:52.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:51.774+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:52.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:51.774+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:52.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:51.774+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:52.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:51.774+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:53.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:52.782+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:53.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:52.782+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:53.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:52.782+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:53.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:52.783+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:53.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:52.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:53.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:52.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:53.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:52.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:53.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:52.835+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:53.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:52.837+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:53.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:52.837+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:53.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:52.837+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:53.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:52.838+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:53.826 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:53.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:53.827 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:53.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:53.827 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:53.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:53.827 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:53.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:53.827 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:53.793+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:53.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:53.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:53.825+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:53.864+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:53.864+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:53.864+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:53.864+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:55.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:54.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:55.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:54.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:54.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:54.752+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:54.905+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:54.905+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:54.905+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:55.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:54.906+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:54.852+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:54.852+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:54.852+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:54.853+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:56.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:02:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:02:55.983+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 15327 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:02:56.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:55.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:55.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:55.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:55.917+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:55.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:55.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:55.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:55.867+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:56.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:55.744+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:56.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:55.744+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:56.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:55.744+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:56.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:55.745+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:57.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:56.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:57.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:56.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:57.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:56.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:56.900+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:57.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:56.713+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:57.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:56.713+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:57.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:56.713+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:57.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:56.714+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:57.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:56.911+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:57.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:56.911+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:57.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:56.911+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:57.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:56.912+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:57.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:57.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:57.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:57.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:57.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:57.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:57.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:57.667+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:57.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:57.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:57.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:57.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:57.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:57.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:57.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:57.888+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:58.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:57.958+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:57.958+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:57.958+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:57.959+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:58.948 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:58.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:58.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:58.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:58.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:58.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:58.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:58.669+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:58.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:58.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:58.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:58.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:58.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:58.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:58.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:58.895+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:59.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:58.946+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:59.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:58.946+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:59.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:58.946+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:58.947+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:59.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:59.932+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:59.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:59.932+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:59.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:59.932+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:02:59.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:02:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:02:59.933+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:02:59.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:59.685+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:59.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:59.685+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:59.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:59.685+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:02:59.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:02:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:02:59.686+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:02:59.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:59.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:59.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:59.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:59.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:59.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:02:59.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:02:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:02:59.933+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:00.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:00.720+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:00.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:00.720+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:00.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:00.720+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:00.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:00.721+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:00.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:00.957+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:00.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:00.957+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:00.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:00.957+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:00.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:00.957+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:00.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:00.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:00.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:00.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:00.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:00.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:00.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:00.953+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:01.260 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:03:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:03:00.984+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 15415 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:03:02.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:01.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:02.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:01.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:01.746+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:01.746+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:01.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:01.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:01.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:01.960+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:02.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:01.931+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:02.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:01.931+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:02.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:01.931+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:02.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:01.932+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:02.761+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:02.761+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:02.761+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:02.762+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:02.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:02.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:02.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:03.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:02.960+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:03.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:02.981+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:03.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:02.981+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:03.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:02.981+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:03.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:02.982+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:03.761+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:03.761+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:03.761+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:03.762+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:03.959+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:03.959+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:03.959+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:03.960+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:04.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:03.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:04.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:03.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:04.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:03.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:04.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:03.925+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:04.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:04.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:04.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:04.812+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:04.908+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:04.908+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:04.908+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:04.909+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:05.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:05.010+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:05.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:05.010+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:05.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:05.010+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:05.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:05.011+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:06.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:03:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:03:05.986+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 15500 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:03:06.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:06.000+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:06.000+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:06.000+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:06.001+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:05.763+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:05.763+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:06.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:05.763+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:06.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:05.763+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:05.956+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:05.956+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:05.956+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:05.957+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:06.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:06.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:06.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:06.749+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:07.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:06.956+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:07.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:06.956+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:07.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:06.956+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:07.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:06.956+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:07.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:06.968+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:07.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:06.968+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:07.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:06.968+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:07.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:06.969+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:07.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:07.716+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:07.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:07.716+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:07.995 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:07.716+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:07.995 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:07.717+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:07.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:07.907+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:07.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:07.907+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:07.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:07.907+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:07.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:07.908+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:08.441 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:07.992+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:08.441 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:07.992+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:08.441 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:07.992+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:08.442 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:07.993+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:09.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:08.739+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:09.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:08.739+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:09.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:08.739+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:09.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:08.739+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:09.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:08.908+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:09.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:08.908+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:09.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:08.908+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:09.037 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:08.908+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:09.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:09.033+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:09.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:09.033+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:09.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:09.033+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:09.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:09.034+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:10.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:09.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:10.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:09.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:10.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:09.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:10.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:09.728+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:10.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:09.912+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:10.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:09.912+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:10.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:09.912+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:10.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:09.913+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:10.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:10.027+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:10.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:10.027+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:10.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:10.027+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:10.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:10.028+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:10.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:10.704+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:10.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:10.704+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:10.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:10.704+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:10.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:10.705+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:11.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:03:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:03:10.988+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 15589 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:03:11.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:10.943+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:11.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:10.943+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:11.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:10.943+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:11.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:10.943+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:11.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:11.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:11.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:11.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:11.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:11.038+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:11.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:11.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:11.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:11.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:11.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:11.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:11.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:11.672+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:11.900+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:11.900+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:11.900+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:11.901+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:12.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:12.072+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:12.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:12.072+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:12.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:12.072+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:12.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:12.072+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:13.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:12.715+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:13.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:12.715+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:13.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:12.715+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:13.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:12.715+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:13.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:12.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:13.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:12.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:13.124 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:12.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:13.124 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:12.878+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:13.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:13.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:13.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:13.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:13.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:13.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:13.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:13.121+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:13.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:13.745+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:13.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:13.745+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:13.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:13.745+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:13.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:13.745+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:13.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:14.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:13.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:14.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:13.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:14.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:13.897+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:14.091+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:14.091+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:14.091+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:14.091+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:15.068 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:14.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:15.068 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:14.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:15.069 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:14.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:15.069 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:14.781+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:15.069 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:14.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:15.069 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:14.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:15.069 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:14.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:15.069 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:14.929+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:15.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:15.066+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:15.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:15.066+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:15.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:15.066+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:15.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:15.067+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:16.074 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:03:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:03:15.989+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 15674 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:03:16.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:15.932+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:16.075 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:15.932+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:16.075 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:15.932+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:16.075 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:15.932+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:16.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:15.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:16.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:15.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:16.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:15.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:16.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:15.800+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:16.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:16.072+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:16.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:16.072+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:16.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:16.072+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:16.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:16.073+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:17.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:16.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:17.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:16.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:17.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:16.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:17.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:16.841+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:17.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:16.958+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:17.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:16.958+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:17.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:16.958+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:17.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:16.959+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:17.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:17.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:17.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:17.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:17.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:17.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:17.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:17.077+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:18.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:17.999+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:18.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:17.999+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:17.999+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:17.999+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:17.866+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:17.866+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:17.866+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:17.867+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:18.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:18.057+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:18.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:18.057+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:18.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:18.057+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:18.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:18.058+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:19.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:19.066+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:19.066+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:19.066+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:19.067+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:18.849+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:18.849+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:18.849+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:19.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:18.850+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:19.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:19.014+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:19.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:19.014+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:19.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:19.014+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:19.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:19.015+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:20.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:19.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:20.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:19.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:20.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:19.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:20.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:19.879+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:20.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:20.062+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:20.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:20.062+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:20.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:20.062+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:20.097 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:20.063+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:20.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:20.093+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:20.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:20.093+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:20.093+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:20.094+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:21.089 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:03:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:03:20.991+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 15762 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:03:21.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:20.873+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:21.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:20.873+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:21.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:20.873+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:21.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:20.874+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:21.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:21.126+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:21.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:21.126+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:21.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:21.126+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:21.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:21.126+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:21.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:21.086+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:21.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:21.086+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:21.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:21.086+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:21.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:21.087+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:22.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:21.833+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:22.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:21.833+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:22.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:21.833+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:22.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:21.834+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:22.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:22.038+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:22.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:22.038+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:22.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:22.038+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:22.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:22.039+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:22.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:22.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:22.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:22.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:22.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:22.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:22.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:22.132+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:23.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:22.784+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:23.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:22.784+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:23.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:22.784+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:23.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:22.785+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:23.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:23.091+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:23.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:23.091+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:23.091+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:23.092+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:23.069+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:23.069+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:23.069+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:23.070+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:23.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:23.789+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:23.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:23.789+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:23.789+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:23.789+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:24.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:24.119+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:24.119+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:24.119+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:24.119+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:24.101+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:24.101+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:24.101+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:24.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:24.101+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:25.101 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:24.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:25.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:24.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:25.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:24.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:25.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:24.764+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:25.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:25.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:25.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:25.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:25.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:25.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:25.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:25.120+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:25.100+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:25.100+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:25.100+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:25.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:25.100+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:26.072 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:03:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:03:25.993+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 15847 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:03:26.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:25.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:26.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:25.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:26.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:25.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:26.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:25.810+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:26.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:26.058+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:26.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:26.058+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:26.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:26.058+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:26.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:26.058+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:26.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:26.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:26.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:26.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:26.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:26.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:26.374 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:26.071+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:27.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:26.860+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:26.860+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:26.860+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:26.860+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:27.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:27.073+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:27.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:27.073+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:27.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:27.073+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:27.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:27.073+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:27.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:27.010+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:27.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:27.010+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:27.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:27.010+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:27.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:27.010+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:28.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:27.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:28.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:27.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:27.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:27.888+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:27.974+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:27.974+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:27.974+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:27.974+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:28.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:28.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:28.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:28.077+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:29.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:28.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:29.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:28.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:29.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:28.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:29.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:28.939+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:29.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:28.936+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:29.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:28.936+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:29.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:28.936+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:29.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:28.936+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:29.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:29.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:29.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:29.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:29.110+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:30.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:29.979+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:30.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:29.979+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:29.979+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:29.980+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:30.152+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:30.152+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:30.152+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:30.152+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:30.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:29.950+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:30.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:29.950+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:30.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:29.950+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:30.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:29.951+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:31.007 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:03:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:03:30.995+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 15936 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:03:31.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:30.992+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:31.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:30.992+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:31.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:30.992+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:31.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:30.993+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:31.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:30.909+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:31.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:30.909+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:31.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:30.909+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:31.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:30.910+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:31.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:31.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:31.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:31.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:31.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:31.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:31.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:31.181+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:32.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:31.957+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:32.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:31.957+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:31.957+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:31.958+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:31.973+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:31.973+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:31.973+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:31.974+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:32.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:32.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:32.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:32.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:32.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:32.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:32.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:32.142+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:33.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:32.962+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:33.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:32.962+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:32.962+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:32.962+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:32.965+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:32.965+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:32.965+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:32.966+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:33.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:33.134+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:33.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:33.134+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:33.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:33.134+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:33.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:33.134+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:34.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:34.147+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:34.147+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:34.147+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:34.147+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:34.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:33.949+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:34.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:33.949+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:34.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:33.949+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:34.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:33.950+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:34.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:33.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:34.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:33.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:34.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:33.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:34.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:33.935+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:35.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:34.928+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:35.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:34.928+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:35.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:34.928+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:35.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:34.929+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:35.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:34.978+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:35.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:34.978+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:35.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:34.978+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:35.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:34.978+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:35.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:35.196+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:35.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:35.196+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:35.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:35.196+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:35.197+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:36.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:03:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:03:35.996+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 16021 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:03:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:35.942+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:35.942+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:35.942+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:35.942+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:36.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:35.978+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:36.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:35.978+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:36.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:35.978+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:36.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:35.979+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:36.148+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:36.148+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:36.148+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:36.149+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:37.112+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:37.112+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:37.112+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:37.113+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:36.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:36.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:36.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:37.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:36.962+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:37.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:36.946+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:37.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:36.946+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:37.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:36.946+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:37.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:36.946+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:38.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:38.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:38.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:38.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:38.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:38.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:38.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:38.114+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:38.219 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:37.945+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:38.220 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:37.945+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:38.220 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:37.945+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:38.220 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:37.946+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:38.220 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:37.926+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:38.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:37.926+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:38.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:37.926+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:38.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:37.926+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:39.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:38.973+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:39.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:38.973+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:39.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:38.973+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:39.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:38.974+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:39.099+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:39.099+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:39.099+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:39.100+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:39.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:38.959+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:39.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:38.959+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:39.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:38.959+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:39.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:38.960+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:40.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:39.913+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:40.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:39.913+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:40.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:39.913+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:40.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:39.913+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:40.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:39.991+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:40.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:39.991+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:40.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:39.991+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:40.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:39.991+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:40.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:40.125+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:40.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:40.125+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:40.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:40.125+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:40.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:40.126+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:41.014 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:03:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:03:40.998+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 16109 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:03:41.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:40.941+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:41.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:40.941+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:41.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:40.941+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:41.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:40.942+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:41.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:40.987+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:41.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:40.987+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:41.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:40.987+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:41.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:40.988+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:41.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:41.162+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:41.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:41.162+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:41.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:41.162+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:41.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:41.162+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:42.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:42.172+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:42.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:42.172+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:42.172+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:42.173+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:41.954+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:41.954+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:41.954+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:42.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:41.955+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:42.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:41.961+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:42.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:41.961+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:42.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:41.961+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:42.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:41.962+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:43.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:42.927+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:43.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:42.927+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:43.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:42.927+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:43.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:42.927+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:43.211 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:42.968+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:43.211 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:42.968+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:43.211 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:42.968+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:43.211 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:42.969+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:43.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:43.208+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:43.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:43.208+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:43.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:43.208+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:43.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:43.208+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:44.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:43.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:44.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:43.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:44.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:43.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:44.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:43.883+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:44.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:43.951+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:44.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:43.951+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:44.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:43.951+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:44.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:43.951+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:44.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:44.171+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:44.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:44.171+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:44.171+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:44.172+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:45.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:44.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:45.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:44.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:45.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:44.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:45.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:44.881+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:45.144 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:44.931+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:45.144 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:44.931+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:45.144 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:44.931+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:45.145 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:44.932+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:45.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:45.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:45.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:45.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:45.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:45.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:45.142+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:46.155 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:03:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:03:46.000+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 16194 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:03:46.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:45.874+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:46.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:45.874+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:46.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:45.874+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:46.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:45.875+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:46.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:45.895+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:46.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:45.895+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:46.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:45.895+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:46.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:45.896+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:46.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:46.153+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:46.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:46.153+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:46.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:46.153+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:46.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:46.153+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:47.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:46.908+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:47.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:46.908+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:47.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:46.908+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:47.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:46.908+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:47.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:46.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:47.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:46.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:47.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:46.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:47.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:46.929+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:47.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:47.200+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:47.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:47.200+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:47.200+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:47.201+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:48.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:47.883+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:48.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:47.883+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:48.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:47.883+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:48.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:47.884+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:48.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:47.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:48.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:47.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:48.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:47.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:48.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:47.935+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:48.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:48.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:48.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:48.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:48.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:48.171+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:49.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:48.871+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:49.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:48.871+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:49.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:48.871+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:49.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:48.872+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:49.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:48.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:49.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:48.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:49.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:48.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:49.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:48.897+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:49.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:49.156+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:49.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:49.156+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:49.156+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:49.158+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:50.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:49.838+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:50.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:49.838+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:50.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:49.838+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:50.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:49.839+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:50.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:49.932+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:50.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:49.932+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:50.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:49.932+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:50.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:49.932+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:50.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:50.202+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:50.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:50.202+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:50.202+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:50.202+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:51.173 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:03:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:03:51.002+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 16283 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:03:51.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:50.861+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:51.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:50.861+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:51.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:50.861+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:51.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:50.862+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:51.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:50.926+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:51.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:50.926+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:51.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:50.926+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:51.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:50.926+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:51.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:51.171+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:51.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:51.171+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:51.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:51.171+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:51.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:51.172+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:52.196 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:51.863+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:52.196 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:51.863+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:52.196 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:51.863+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:52.196 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:51.864+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:52.196 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:51.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:52.196 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:51.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:52.197 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:51.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:52.197 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:51.890+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:52.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:52.193+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:52.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:52.193+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:52.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:52.193+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:52.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:52.194+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:53.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:52.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:53.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:52.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:53.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:52.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:53.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:52.873+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:53.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:52.930+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:53.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:52.930+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:53.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:52.930+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:53.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:52.930+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:53.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:53.177+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:53.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:53.177+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:53.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:53.177+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:53.178+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:54.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:53.830+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:54.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:53.830+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:54.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:53.830+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:54.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:53.831+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:54.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:53.914+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:54.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:53.914+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:54.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:53.914+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:54.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:53.915+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:54.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:54.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:54.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:54.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:54.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:54.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:54.171+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:55.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:54.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:55.195 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:54.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:55.195 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:54.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:55.195 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:54.855+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:55.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:54.963+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:55.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:54.963+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:55.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:54.963+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:55.196 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:54.964+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:55.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:55.192+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:55.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:55.192+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:55.192+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:55.193+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:56.152 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:03:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:03:56.004+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 16368 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:03:56.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:55.898+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:56.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:55.898+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:56.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:55.898+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:56.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:55.899+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:56.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:55.978+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:56.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:55.978+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:56.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:55.978+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:56.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:55.978+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:56.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:56.150+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:56.150+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:56.150+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:56.151+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:57.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:56.860+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:57.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:56.860+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:57.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:56.860+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:57.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:56.860+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:57.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:57.023+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:57.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:57.023+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:57.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:57.023+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:57.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:57.023+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:57.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:57.158+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:57.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:57.158+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:57.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:57.158+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:57.159+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:58.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:58.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:58.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:58.142+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:57.895+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:58.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:57.895+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:58.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:57.895+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:58.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:57.895+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:58.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:58.043+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:58.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:58.043+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:58.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:58.043+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:58.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:58.044+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:59.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:58.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:59.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:58.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:59.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:58.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:03:59.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:58.875+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:03:59.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:59.014+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:59.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:59.014+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:59.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:59.014+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:03:59.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:03:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:03:59.015+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:03:59.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:59.138+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:59.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:59.138+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:59.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:59.138+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:03:59.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:03:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:03:59.139+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:00.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:00.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:00.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:00.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:00.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:00.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:00.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:00.014+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:00.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:59.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:00.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:59.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:00.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:59.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:00.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:03:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:03:59.903+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:00.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:00.117+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:00.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:00.117+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:00.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:00.117+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:00.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:00.117+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:01.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:00.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:01.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:00.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:01.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:00.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:01.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:00.931+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:01.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:00.969+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:01.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:00.969+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:01.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:00.969+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:01.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:00.970+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:01.262 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:04:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:04:01.006+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 16456 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:04:01.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:01.072+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:01.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:01.072+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:01.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:01.072+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:01.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:01.073+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:02.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:01.927+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:02.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:01.927+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:01.927+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:01.927+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:01.885+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:01.885+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:01.885+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:01.885+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:02.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:02.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:02.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:02.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:02.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:02.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:02.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:02.087+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:02.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:02.921+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:02.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:02.921+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:02.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:02.921+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:02.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:02.922+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:03.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:02.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:03.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:02.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:03.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:02.928+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:03.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:02.928+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:03.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:03.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:03.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:03.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:03.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:03.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:03.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:03.137+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:04.159+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:04.159+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:04.159+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:04.160+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:03.935+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:03.935+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:03.935+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:03.935+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:04.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:03.949+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:04.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:03.949+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:04.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:03.949+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:04.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:03.950+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:05.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:04.891+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:05.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:04.891+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:05.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:04.891+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:05.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:04.892+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:05.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:04.922+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:05.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:04.922+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:05.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:04.922+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:05.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:04.923+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:05.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:05.175+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:05.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:05.175+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:05.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:05.175+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:05.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:05.176+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:06.171 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:04:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:04:06.008+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 16541 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:04:06.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:05.852+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:06.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:05.852+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:06.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:05.852+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:06.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:05.852+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:06.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:05.888+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:06.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:05.888+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:06.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:05.888+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:06.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:05.889+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:06.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:06.169+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:06.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:06.169+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:06.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:06.169+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:06.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:06.170+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:07.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:06.850+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:07.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:06.850+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:07.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:06.850+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:07.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:06.851+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:07.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:06.900+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:07.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:06.900+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:07.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:06.900+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:07.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:06.901+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:07.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:07.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:07.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:07.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:07.181+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:08.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:07.870+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:08.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:07.870+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:08.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:07.870+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:08.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:07.871+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:08.136 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:07.911+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:08.136 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:07.911+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:08.136 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:07.911+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:08.136 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:07.912+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:08.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:08.133+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:08.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:08.133+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:08.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:08.133+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:08.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:08.133+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:09.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:08.886+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:09.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:08.886+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:09.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:08.886+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:09.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:08.887+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:09.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:08.873+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:09.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:08.873+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:09.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:08.873+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:09.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:08.873+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:09.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:09.159+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:09.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:09.159+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:09.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:09.159+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:09.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:09.160+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:10.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:09.905+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:10.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:09.905+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:10.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:09.905+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:10.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:09.906+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:10.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:09.842+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:10.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:09.842+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:10.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:09.842+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:10.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:09.843+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:10.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:10.146+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:10.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:10.146+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:10.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:10.146+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:10.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:10.147+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:11.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:10.908+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:11.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:10.908+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:11.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:10.908+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:11.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:10.909+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:11.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:10.827+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:11.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:10.827+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:11.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:10.827+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:11.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:10.828+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:11.359 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:04:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:04:11.009+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 16630 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:04:11.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:11.151+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:11.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:11.151+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:11.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:11.151+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:11.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:11.151+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:12.143 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:11.852+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:12.144 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:11.852+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:12.144 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:11.852+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:12.144 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:11.852+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:12.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:11.892+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:12.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:11.892+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:12.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:11.892+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:12.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:11.892+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:12.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:12.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:12.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:12.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:12.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:12.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:12.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:12.142+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:12.914+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:12.914+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:12.914+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:12.915+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:12.878+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:12.878+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:12.878+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:13.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:12.879+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:13.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:13.112+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:13.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:13.112+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:13.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:13.112+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:13.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:13.112+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:13.894+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:14.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:13.894+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:13.894+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:13.894+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:14.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:13.922+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:14.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:13.922+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:14.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:13.922+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:14.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:13.923+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:14.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:14.124+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:14.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:14.124+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:14.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:14.124+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:14.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:14.124+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:15.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:14.899+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:14.899+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:14.899+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:14.899+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:15.145+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:15.145+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:15.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:15.145+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:15.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:15.146+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:14.930+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:14.930+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:14.930+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:14.931+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:16.130 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:04:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:04:16.011+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 16715 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:04:16.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:15.853+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:16.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:15.853+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:16.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:15.853+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:16.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:15.853+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:16.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:15.975+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:16.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:15.975+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:16.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:15.975+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:16.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:15.976+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:16.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:16.128+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:16.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:16.128+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:16.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:16.128+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:16.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:16.129+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:17.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:16.950+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:16.950+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:16.950+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:16.951+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:17.104+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:17.104+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:17.104+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:17.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:17.104+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:17.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:16.863+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:17.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:16.863+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:17.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:16.863+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:17.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:16.864+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:18.111 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:17.943+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:18.111 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:17.943+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:18.111 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:17.943+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:18.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:17.943+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:18.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:17.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:18.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:17.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:18.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:17.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:18.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:17.849+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:18.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:18.108+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:18.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:18.108+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:18.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:18.108+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:18.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:18.109+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:19.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:18.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:19.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:18.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:19.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:18.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:19.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:18.819+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:19.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:18.966+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:19.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:18.966+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:19.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:18.966+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:19.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:18.967+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:19.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:19.108+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:19.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:19.108+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:19.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:19.108+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:19.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:19.109+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:20.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:19.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:20.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:19.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:20.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:19.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:20.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:19.793+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:20.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:19.966+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:20.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:19.966+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:20.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:19.966+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:20.085 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:19.966+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:20.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:20.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:20.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:20.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:20.081+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:21.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:20.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:21.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:20.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:21.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:20.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:21.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:20.783+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:21.260 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:04:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:04:21.012+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 16804 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:04:21.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:21.095+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:21.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:21.095+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:21.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:21.095+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:21.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:21.096+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:21.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:21.001+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:21.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:21.001+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:21.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:21.001+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:21.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:21.001+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:22.038 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:21.745+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:22.038 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:21.745+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:22.038 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:21.745+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:22.038 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:21.745+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:22.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:22.034+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:22.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:22.034+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:22.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:22.034+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:22.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:22.035+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:22.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:22.102+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:22.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:22.102+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:22.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:22.102+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:22.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:22.103+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:22.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:22.697+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:22.697+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:22.697+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:22.698+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:23.049+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:23.049+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:23.050+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:23.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:23.050+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:23.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:23.055+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:23.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:23.055+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:23.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:23.055+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:23.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:23.056+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:23.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:23.688+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:23.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:23.688+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:23.688+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:23.688+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:24.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:24.100+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:24.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:24.100+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:24.337 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:24.100+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:24.337 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:24.101+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:24.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:24.024+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:24.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:24.024+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:24.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:24.024+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:24.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:24.025+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:24.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:24.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:24.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:24.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:24.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:24.727+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:25.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:25.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:25.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:25.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:25.102+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:25.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:25.041+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:25.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:25.041+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:25.041+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:25.042+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:26.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:25.725+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:26.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:25.725+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:26.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:25.725+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:26.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:25.725+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:26.372 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:04:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:04:26.015+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 16889 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:04:26.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:26.108+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:26.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:26.108+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:26.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:26.108+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:26.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:26.109+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:26.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:26.049+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:26.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:26.049+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:26.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:26.049+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:26.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:26.049+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:27.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:26.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:27.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:26.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:27.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:26.753+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:27.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:26.754+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:27.081 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:27.078+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:27.081 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:27.078+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:27.082 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:27.078+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:27.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:27.079+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:27.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:27.099+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:27.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:27.099+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:27.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:27.099+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:27.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:27.100+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:28.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:27.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:28.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:27.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:28.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:27.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:28.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:27.728+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:28.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:28.087+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:28.087+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:28.087+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:28.088+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:28.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:28.135+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:28.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:28.135+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:28.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:28.135+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:28.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:28.136+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:29.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:28.766+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:29.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:28.766+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:29.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:28.766+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:29.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:28.767+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:29.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:29.083+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:29.083+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:29.083+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:29.084+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:29.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:29.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:29.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:29.088+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:29.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:29.730+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:29.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:29.730+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:29.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:29.730+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:29.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:29.730+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:30.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:30.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:30.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:30.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:30.089+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:30.083+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:30.083+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:30.083+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:30.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:30.083+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:30.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:30.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:30.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:30.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:30.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:30.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:30.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:30.728+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:31.355 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:04:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:04:31.017+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 16978 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:04:31.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:31.076+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:31.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:31.076+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:31.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:31.076+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:31.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:31.077+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:31.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:31.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:31.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:31.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:31.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:31.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:31.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:31.037+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:31.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:31.695+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:31.695+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:31.695+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:31.696+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:32.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:32.073+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:32.073+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:32.073+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:32.073+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:32.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:32.080+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:32.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:32.080+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:32.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:32.080+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:32.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:32.080+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:32.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:32.700+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:32.700+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:32.700+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:32.701+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:33.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:33.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:33.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:33.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:33.103+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:33.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:33.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:33.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:33.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:33.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:33.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:33.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:33.077+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:33.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:33.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:33.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:33.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:33.741+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:34.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:34.116+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:34.116+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:34.116+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:34.117+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:34.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:34.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:34.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:34.038+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:35.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:34.717+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:35.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:34.717+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:35.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:34.717+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:35.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:34.718+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:35.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:35.000+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:35.000+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:35.000+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:35.001+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:35.152+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:35.152+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:35.152+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:35.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:35.153+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:35.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:35.692+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:35.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:35.692+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:35.692+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:35.693+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:36.366 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:04:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:04:36.019+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 17063 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:04:36.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:36.197+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:36.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:36.197+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:36.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:36.197+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:36.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:36.197+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:36.367 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:35.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:36.367 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:35.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:36.368 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:35.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:36.368 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:35.989+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:36.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:36.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:36.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:36.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:36.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:36.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:36.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:36.669+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:37.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:37.148+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:37.148+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:37.148+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:37.149+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:36.958+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:36.958+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:36.958+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:36.959+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:37.696+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:37.696+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:37.696+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:37.697+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:38.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:38.134+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:38.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:38.134+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:38.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:38.134+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:38.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:38.135+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:37.996+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:37.996+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:37.996+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:38.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:37.997+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:38.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:38.677+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:38.677+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:38.677+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:38.677+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:39.168+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:39.168+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:39.168+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:39.169+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:39.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:38.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:39.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:38.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:39.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:38.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:39.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:38.981+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:39.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:39.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:39.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:39.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:39.941+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:39.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:39.717+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:39.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:39.717+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:39.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:39.717+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:39.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:39.717+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:40.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:40.180+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:40.180+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:40.180+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:40.181+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:41.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:40.718+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:41.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:40.718+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:41.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:40.718+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:41.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:40.719+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:41.024 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:40.914+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:41.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:40.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:41.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:40.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:41.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:40.915+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:41.357 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:04:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:04:41.021+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 17152 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:04:41.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:41.184+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:41.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:41.184+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:41.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:41.184+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:41.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:41.185+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:42.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:41.757+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:42.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:41.757+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:42.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:41.757+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:42.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:41.758+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:42.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:41.914+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:42.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:41.914+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:42.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:41.914+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:42.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:41.915+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:42.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:42.159+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:42.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:42.159+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:42.159+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:42.160+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:42.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:42.710+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:42.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:42.710+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:42.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:42.710+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:42.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:42.710+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:42.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:42.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:42.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:42.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:42.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:42.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:42.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:42.933+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:43.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:43.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:43.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:43.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:43.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:43.189+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:43.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:43.700+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:43.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:43.700+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:43.927 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:43.700+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:43.927 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:43.700+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:44.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:43.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:44.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:43.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:44.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:43.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:44.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:43.925+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:44.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:44.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:44.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:44.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:44.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:44.201+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:44.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:44.682+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:44.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:44.682+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:44.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:44.682+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:44.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:44.682+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:45.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:44.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:44.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:44.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:44.942+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:45.683 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:45.229+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:45.683 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:45.229+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:45.683 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:45.229+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:45.684 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:45.230+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:45.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:45.937+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:45.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:45.937+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:45.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:45.937+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:45.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:45.681+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:45.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:45.681+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:45.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:45.681+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:45.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:45.681+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:04:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:04:46.023+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 17237 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:04:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:45.938+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:46.474 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:46.258+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:46.474 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:46.258+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:46.474 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:46.258+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:46.475 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:46.258+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:46.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:46.637+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:46.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:46.637+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:46.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:46.637+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:46.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:46.638+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:47.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:46.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:46.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:46.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:46.981+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:47.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:47.216+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:47.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:47.216+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:47.662 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:47.216+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:47.662 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:47.217+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:47.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:47.659+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:47.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:47.659+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:47.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:47.659+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:47.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:47.660+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:48.198 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:47.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:48.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:47.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:48.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:47.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:48.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:47.940+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:48.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:48.196+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:48.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:48.196+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:48.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:48.196+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:48.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:48.197+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:48.933 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:48.648+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:48.934 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:48.648+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:48.934 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:48.648+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:48.934 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:48.649+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:49.148+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:49.148+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:49.148+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:49.149+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:48.931+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:48.931+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:48.931+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:48.932+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:49.888 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:49.612+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:49.888 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:49.612+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:49.888 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:49.612+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:49.889 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:49.612+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:50.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:49.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:50.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:49.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:50.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:49.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:50.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:49.886+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:50.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:50.142+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:50.142+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:50.142+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:50.143+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:50.875 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:50.594+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:50.876 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:50.594+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:50.876 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:50.594+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:50.876 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:50.594+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:51.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:04:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:04:51.025+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 17326 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:04:51.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:51.110+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:51.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:51.110+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:51.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:51.110+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:51.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:51.111+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:50.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:50.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:50.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:51.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:50.874+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:51.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:51.851+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:51.851+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:51.851+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:51.851+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:51.615+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:51.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:51.615+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:51.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:51.615+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:51.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:51.616+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:52.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:52.117+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:52.117+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:52.117+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:52.118+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:52.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:52.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:52.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:52.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:52.858+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:52.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:52.639+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:52.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:52.639+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:52.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:52.639+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:52.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:52.640+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:53.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:53.155+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:53.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:53.155+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:53.155+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:53.155+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:53.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:53.688+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:53.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:53.688+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:53.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:53.688+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:53.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:53.688+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:54.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:53.908+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:54.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:53.908+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:54.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:53.908+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:54.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:53.909+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:54.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:54.167+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:54.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:54.167+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:54.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:54.167+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:54.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:54.168+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:54.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:54.689+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:54.689+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:54.689+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:54.689+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:54.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:54.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:54.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:54.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:54.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:54.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:54.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:54.901+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:55.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:55.208+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:55.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:55.208+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:55.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:55.208+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:55.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:55.209+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:56.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:55.732+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:56.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:55.732+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:56.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:55.732+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:56.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:55.733+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:56.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:55.929+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:56.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:55.929+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:56.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:55.929+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:56.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:55.930+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:56.375 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:04:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:04:56.027+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 17411 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:04:56.376 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:56.238+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:56.376 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:56.238+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:56.376 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:56.238+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:56.376 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:56.239+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:57.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:56.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:57.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:56.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:57.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:56.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:57.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:56.958+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:57.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:56.767+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:57.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:56.767+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:57.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:56.767+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:57.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:56.768+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:57.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:57.219+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:57.219+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:57.219+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:57.220+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:57.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:57.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:57.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:57.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:57.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:57.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:57.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:57.720+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:58.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:57.971+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:58.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:57.971+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:58.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:57.971+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:58.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:57.972+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:58.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:58.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:58.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:58.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:58.265+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:59.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:58.750+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:59.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:58.750+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:59.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:58.750+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:04:59.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:58.751+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:04:59.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:59.013+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:59.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:59.013+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:59.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:59.013+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:04:59.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:04:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:04:59.013+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:04:59.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:59.265+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:59.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:59.265+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:59.265+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:04:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:04:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:04:59.266+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:00.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:59.769+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:00.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:59.769+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:00.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:59.769+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:00.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:04:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:04:59.770+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:00.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:00.008+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:00.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:00.008+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:00.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:00.008+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:00.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:00.009+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:00.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:00.265+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:00.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:00.265+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:00.265+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:00.266+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:01.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:00.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:01.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:00.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:01.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:00.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:01.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:00.783+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:01.032 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:00.964+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:01.032 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:00.964+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:01.033 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:00.964+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:01.033 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:00.965+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:01.358 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:05:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:05:01.029+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 17499 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:05:01.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:01.265+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:01.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:01.265+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:01.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:01.265+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:01.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:01.266+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:02.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:01.814+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:01.814+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:01.814+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:01.814+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:01.919+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:01.919+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:01.919+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:02.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:01.919+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:02.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:02.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:02.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:02.307+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:02.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:02.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:02.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:02.915+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:02.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:02.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:02.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:02.810+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:03.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:03.342+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:03.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:03.342+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:03.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:03.342+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:03.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:03.342+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:03.853+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:03.853+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:03.853+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:03.853+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:03.929+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:03.929+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:03.929+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:03.930+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:04.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:04.356+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:04.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:04.356+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:04.356+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:04.357+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:04.808+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:04.808+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:04.808+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:04.808+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:04.952+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:04.952+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:04.952+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:04.953+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:05.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:05.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:05.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:05.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:05.366+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:06.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:05:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:05:06.030+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 17584 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:05:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:05.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:05.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:05.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:05.916+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:05.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:05.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:06.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:05.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:06.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:05.841+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:06.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:06.334+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:06.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:06.334+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:06.334+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:06.335+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:07.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:06.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:06.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:06.792+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:06.792+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:06.942+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:06.942+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:06.942+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:06.943+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:07.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:07.290+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:07.290+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:07.290+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:07.291+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:08.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:07.978+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:08.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:07.978+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:07.978+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:07.979+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:07.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:07.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:07.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:07.791+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:08.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:08.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:08.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:08.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:08.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:08.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:08.265+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:09.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:08.816+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:08.816+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:08.816+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:08.817+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:08.968+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:08.968+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:08.968+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:08.968+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:09.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:09.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:09.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:09.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:09.229+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:10.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:09.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:10.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:09.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:10.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:09.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:10.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:09.816+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:10.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:09.973+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:10.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:09.973+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:10.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:09.973+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:10.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:09.974+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:10.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:10.232+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:10.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:10.232+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:10.232+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:10.232+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:11.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:10.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:11.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:10.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:11.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:10.815+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:11.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:10.816+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:11.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:10.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:11.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:10.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:11.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:10.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:11.037 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:10.960+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:11.359 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:05:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:05:11.033+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 17674 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:05:11.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:11.184+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:11.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:11.184+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:11.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:11.184+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:11.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:11.184+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:11.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:11.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:11.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:12.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:11.859+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:11.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:11.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:11.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:11.929+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:12.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:12.213+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:12.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:12.213+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:12.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:12.213+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:12.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:12.213+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:13.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:12.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:13.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:12.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:13.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:12.809+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:13.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:12.810+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:13.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:12.971+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:13.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:12.971+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:13.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:12.971+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:13.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:12.972+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:13.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:13.172+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:13.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:13.172+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:13.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:13.172+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:13.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:13.173+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:13.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:13.808+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:13.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:13.808+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:13.808+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:13.809+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:14.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:14.169+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:14.169+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:14.169+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:14.169+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:14.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:13.976+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:14.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:13.976+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:14.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:13.976+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:14.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:13.977+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:15.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:14.763+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:15.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:14.763+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:15.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:14.763+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:15.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:14.764+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:15.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:14.963+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:15.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:14.963+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:15.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:14.963+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:15.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:14.964+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:15.429 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:15.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:15.429 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:15.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:15.429 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:15.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:15.429 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:15.175+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:16.038 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:15.756+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:16.038 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:15.756+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:16.038 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:15.756+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:16.039 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:15.757+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:16.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:15.995+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:16.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:15.995+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:16.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:15.995+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:16.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:15.996+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:16.372 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:05:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:05:16.035+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 17760 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:05:16.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:16.211+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:16.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:16.211+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:16.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:16.211+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:16.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:16.212+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:17.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:16.993+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:16.993+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:16.993+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:16.993+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:16.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:16.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:16.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:17.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:16.748+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:17.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:17.247+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:17.247+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:17.247+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:17.247+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:18.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:17.999+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:17.999+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:17.999+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:18.000+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:17.760+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:17.760+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:17.760+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:18.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:17.761+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:18.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:18.266+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:18.266+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:18.266+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:18.266+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:18.985+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:18.985+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:18.985+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:18.986+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:18.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:18.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:19.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:18.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:19.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:18.752+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:19.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:19.248+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:19.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:19.248+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:19.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:19.248+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:19.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:19.249+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:20.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:19.794+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:20.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:19.794+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:20.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:19.794+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:20.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:19.795+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:20.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:20.018+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:20.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:20.018+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:20.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:20.018+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:20.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:20.019+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:20.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:20.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:20.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:20.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:20.222+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:21.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:20.762+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:21.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:20.762+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:21.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:20.762+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:21.025 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:20.763+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:21.361 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:05:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:05:21.037+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 17849 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:05:21.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:21.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:21.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:21.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:21.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:21.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:21.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:21.252+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:21.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:21.022+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:21.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:21.022+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:21.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:21.022+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:21.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:21.023+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:22.066 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:21.741+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:22.067 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:21.741+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:22.067 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:21.741+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:22.067 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:21.742+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:22.338 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:22.277+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:22.339 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:22.277+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:22.339 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:22.277+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:22.339 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:22.278+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:22.339 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:22.064+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:22.339 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:22.064+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:22.340 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:22.064+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:22.340 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:22.065+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:22.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T20:05:22.335+0000 7fe5c215f640 0 monclient(hunting): authenticate timed out after 300 2024-09-20T20:05:22.340 INFO:teuthology.orchestra.run.smithi114.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-09-20T20:05:22.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:22.762+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:22.762+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:22.762+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:22.763+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:23.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T20:05:23.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd blacklist ls 2024-09-20T20:05:23.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:23.101+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:23.101+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:23.101+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:23.102+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:23.313+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:23.313+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:23.313+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:23.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:23.313+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:23.470 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T20:05:23.863 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:23.729+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:23.863 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:23.729+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:23.863 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:23.729+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:23.863 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:23.730+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:24.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:24.086+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:24.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:24.086+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:24.086+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:24.086+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:24.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:24.302+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:24.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:24.302+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:24.302+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:24.303+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:25.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:24.771+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:25.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:24.771+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:25.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:24.771+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:25.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:24.771+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:25.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:25.100+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:25.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:25.100+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:25.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:25.100+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:25.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:25.101+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:25.289+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:25.289+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:25.289+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:25.290+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:25.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:25.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:25.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:25.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:25.737+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:26.334 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:05:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:05:26.039+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 17935 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:05:26.335 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:26.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:26.335 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:26.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:26.335 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:26.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:26.335 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:26.078+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:26.705 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:26.332+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:26.706 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:26.332+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:26.706 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:26.332+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:26.706 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:26.333+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:26.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:26.703+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:26.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:26.703+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:26.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:26.703+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:26.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:26.704+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:27.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:27.093+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:27.093+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:27.093+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:27.093+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:27.308+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:27.308+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:27.308+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:27.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:27.308+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:28.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:27.732+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:28.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:27.732+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:28.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:27.732+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:28.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:27.733+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:28.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:28.277+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:28.277+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:28.277+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:28.278+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:28.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:28.132+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:28.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:28.132+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:28.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:28.132+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:28.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:28.133+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:28.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:28.698+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:28.698+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:28.698+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:28.698+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:29.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:29.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:29.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:29.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:29.265+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:29.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:29.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:29.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:29.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:29.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:29.110+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:30.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:29.745+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:30.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:29.745+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:30.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:29.745+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:30.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:29.746+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:30.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:30.297+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:30.297+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:30.297+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:30.298+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:30.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:30.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:30.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:30.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:30.121+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:30.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:30.704+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:30.704+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:30.704+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:30.704+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:31.358 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:05:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:05:31.041+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 18026 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:05:31.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:31.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:31.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:31.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:31.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:31.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:31.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:31.272+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:31.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:31.096+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:31.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:31.096+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:31.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:31.096+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:31.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:31.097+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:31.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:31.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:31.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:31.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:31.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:31.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:31.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:31.669+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:32.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:32.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:32.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:32.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:32.308+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:32.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:32.079+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:32.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:32.079+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:32.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:32.079+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:32.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:32.079+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:32.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:32.692+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:32.692+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:32.692+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:32.693+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:33.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:33.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:33.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:33.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:33.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:33.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:33.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:33.051+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:33.589 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:33.304+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:33.590 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:33.304+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:33.590 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:33.304+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:33.590 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:33.304+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:33.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:33.649+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:33.649+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:33.649+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:33.650+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:34.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:34.071+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:34.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:34.071+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:34.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:34.071+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:34.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:34.072+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:34.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:34.343+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:34.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:34.343+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:34.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:34.343+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:34.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:34.343+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:34.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:34.658+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:34.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:34.658+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:34.658+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:34.659+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:35.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:35.055+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:35.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:35.055+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:35.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:35.055+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:35.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:35.056+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:35.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:35.382+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:35.638 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:35.382+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:35.638 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:35.382+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:35.638 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:35.383+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:35.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:35.635+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:35.635+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:35.635+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:35.636+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:36.369 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:05:36.043+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 18112 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:05:36.369 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:36.035+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:36.369 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:36.035+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:36.370 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:36.035+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:36.370 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:36.035+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:36.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:36.407+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:36.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:36.407+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:36.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:36.407+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:36.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:36.407+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:36.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:36.669+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:36.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:36.669+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:36.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:36.669+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:36.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:36.670+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:37.381 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:36.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:37.381 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:36.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:37.381 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:36.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:37.381 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:36.989+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:37.640 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:37.378+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:37.640 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:37.378+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:37.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:37.378+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:37.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:37.379+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:37.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:37.638+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:37.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:37.638+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:37.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:37.638+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:37.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:37.639+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:38.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:37.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:37.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:37.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:37.958+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:38.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:38.375+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:38.375+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:38.375+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:38.376+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:38.601+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:38.601+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:38.601+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:38.601+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:39.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:38.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:39.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:38.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:38.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:38.930+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:39.620+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:39.620+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:39.620+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:39.621+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:39.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:39.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:39.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:39.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:39.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:39.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:39.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:39.398+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:40.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:39.912+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:40.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:39.912+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:40.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:39.912+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:40.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:39.913+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:40.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:40.593+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:40.593+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:40.593+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:40.594+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:40.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:40.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:40.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:40.442+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:41.048 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:40.884+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:41.048 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:40.884+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:41.048 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:40.884+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:41.048 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:40.884+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:41.361 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:05:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:05:41.045+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 18201 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:05:41.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:41.563+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:41.563+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:41.563+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:41.563+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:41.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:41.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:41.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:41.487+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:42.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:41.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:42.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:41.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:41.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:41.908+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:42.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:42.521+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:42.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:42.521+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:42.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:42.521+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:42.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:42.522+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:42.871 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:42.576+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:42.871 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:42.576+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:42.871 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:42.576+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:42.871 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:42.577+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:43.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:42.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:43.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:42.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:42.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:42.868+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:43.878 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:43.578+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:43.878 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:43.578+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:43.878 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:43.578+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:43.878 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:43.578+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:43.879 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:43.492+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:43.879 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:43.492+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:43.879 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:43.492+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:43.879 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:43.492+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:44.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:43.875+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:44.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:43.875+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:44.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:43.875+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:14.047021+0000 (oldest deadline 2024-09-20T19:55:34.047021+0000) 2024-09-20T20:05:44.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:43.876+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:44.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:44.445+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:44.445+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:44.445+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:44.446+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:44.918 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:44.575+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:44.918 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:44.575+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:44.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:44.575+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:44.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:44.575+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:45.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:44.916+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:45.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:45.602+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:45.602+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:45.602+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:45.603+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:45.412+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:45.412+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:45.412+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:45.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:45.412+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:05:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:05:46.047+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 18287 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:05:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:45.871+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:46.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:46.569+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:46.569+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:46.569+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:46.569+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:46.376+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:46.376+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:46.376+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:46.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:46.376+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:47.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:46.856+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:47.486 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:47.386+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:47.486 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:47.386+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:47.486 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:47.386+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:47.486 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:47.387+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:47.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:47.577+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:47.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:47.577+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:47.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:47.577+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:47.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:47.578+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:48.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:47.892+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:48.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:48.554+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:48.554+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:48.554+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:17.209315+0000 (oldest deadline 2024-09-20T19:55:37.209315+0000) 2024-09-20T20:05:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:48.555+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:48.373+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:48.373+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:48.373+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:48.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:48.374+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:49.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:48.889+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:49.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:49.517+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:49.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:49.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:49.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:49.397+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:50.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:49.900+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:50.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:50.530+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:50.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:50.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:50.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:50.432+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:51.051 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:50.853+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:51.359 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:05:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:05:51.049+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 18377 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:05:51.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:51.542+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:51.388+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:51.388+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:51.388+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:51.389+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:52.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:51.813+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:52.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:52.561+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:52.368+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:52.368+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:52.368+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:52.369+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:52.824+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:53.591 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:53.338+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:53.591 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:53.338+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:53.591 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:53.338+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:53.591 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:53.338+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:53.595+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:53.831+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:54.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:54.574+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:54.349+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:54.349+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:54.349+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:54.350+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:55.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:54.830+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:55.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:55.538+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:55.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:55.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:55.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:55.398+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:56.087 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:05:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:05:56.052+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 18463 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:05:56.087 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:55.833+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:56.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:56.554+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:56.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:56.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:56.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:56.425+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:57.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:56.790+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:57.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:57.507+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:57.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:57.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:57.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:57.424+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:57.742+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:58.639 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:58.492+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:58.639 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:58.377+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:58.640 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:58.377+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:58.640 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:58.377+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:58.640 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:58.378+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:58.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:58.768+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:05:59.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:05:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:05:59.443+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:05:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:59.333+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:59.333+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:59.333+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:05:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:05:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:05:59.333+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:00.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:05:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:05:59.736+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:00.435 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:00.354+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:06:00.436 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:00.354+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:06:00.436 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:00.354+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T19:55:27.126969+0000 (oldest deadline 2024-09-20T19:55:47.126969+0000) 2024-09-20T20:06:00.436 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:00.355+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:00.433+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:01.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:00.778+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:01.361 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:06:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:06:01.054+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 18552 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:06:01.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:01.417+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:01.375+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:02.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:01.792+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:02.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:02.433+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:02.396+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:02.805+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:03.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:03.481+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:03.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:03.348+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:03.808+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:04.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:04.475+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:04.353+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:05.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:04.795+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:05.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:05.458+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:05.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:05.385+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:05.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:05.795+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:06.373 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:06:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:06:06.056+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 18638 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:06:06.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:06.468+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:06.432+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:07.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:06.775+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:06.775+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:06.775+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:06.776+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:07.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:07.481+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:07.434+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:08.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:07.820+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:07.820+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:07.820+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:07.821+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:08.440+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:08.452+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:09.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:08.796+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:08.796+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:08.796+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:08.797+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:09.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:09.435+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:09.462+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:10.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:09.761+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:10.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:09.761+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:10.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:09.761+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:10.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:09.761+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:10.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:10.397+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:10.421+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:10.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:10.712+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:10.712+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:10.712+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:10.713+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:11.318 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:06:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:06:11.058+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 18728 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:06:11.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:11.450+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:11.403+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:11.403+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:11.403+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:11.404+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:12.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:11.749+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:11.749+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:11.749+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:11.749+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:12.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:12.405+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:12.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:12.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:12.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:12.385+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:13.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:12.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:13.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:12.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:12.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:12.784+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:13.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:13.413+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:13.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:13.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:13.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:13.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:13.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:13.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:13.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:13.389+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:14.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:13.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:13.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:13.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:13.780+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:14.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:14.434+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:14.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:14.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:14.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:14.429+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:15.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:14.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:14.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:14.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:14.812+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:15.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:15.451+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:15.453+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:15.453+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:15.453+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:15.453+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:16.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:06:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:06:16.060+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 18814 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:06:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:15.840+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:15.840+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:15.840+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:15.840+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:16.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:16.477+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:16.462+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:16.462+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:16.462+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:16.463+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:17.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:16.846+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:16.846+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:16.846+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:16.847+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:17.513 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:17.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:17.513 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:17.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:17.513 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:17.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:17.514 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:17.419+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:17.853 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:17.511+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:18.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:17.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:17.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:17.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:17.851+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:18.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:18.541+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:18.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:18.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:18.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:18.385+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:19.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:18.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:18.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:18.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:18.880+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:19.571+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:19.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:19.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:19.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:19.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:19.382+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:20.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:19.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:20.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:19.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:20.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:19.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:20.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:19.896+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:20.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:20.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:20.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:20.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:20.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:20.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:20.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:20.346+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:20.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:20.528+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:21.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:06:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:06:21.062+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 18903 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:06:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:20.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:20.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:20.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:20.931+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:21.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:21.490+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:21.388+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:21.388+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:21.388+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:21.389+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:22.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:21.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:21.935+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:21.935+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:21.935+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:22.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:22.530+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:22.368+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:22.368+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:22.368+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:22.369+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:23.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:22.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:22.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:22.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:22.887+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:23.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:23.494+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:23.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:23.404+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:23.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:23.404+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:23.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:23.404+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:23.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:23.404+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:24.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:23.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:24.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:23.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:24.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:23.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:24.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:23.891+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:24.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:24.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:24.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:24.363+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:24.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:24.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:24.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:24.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:24.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:24.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:24.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:24.542+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:24.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:24.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:24.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:24.871+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:25.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:25.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:25.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:25.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:25.329+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:25.537+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:25.537+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:25.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:25.537+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:25.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:25.538+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:26.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:06:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:06:26.064+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 18989 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:06:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:25.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:25.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:25.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:25.855+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:26.568 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:26.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:26.568 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:26.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:26.568 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:26.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:26.569 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:26.308+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:26.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:26.810+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:26.810+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:26.810+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:26.811+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:26.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:26.580+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:26.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:26.580+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:26.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:26.580+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:26.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:26.581+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:27.625 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:27.286+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:27.625 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:27.286+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:27.625 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:27.286+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:27.625 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:27.287+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:27.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:27.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:27.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:27.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:27.821+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:27.622+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:27.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:27.622+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:27.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:27.622+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:27.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:27.623+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:28.656 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:28.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:28.656 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:28.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:28.656 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:28.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:28.656 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:28.335+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:28.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:28.653+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:28.653+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:28.653+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:28.654+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:28.777+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:28.777+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:28.777+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:28.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:28.777+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:29.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:29.380+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:29.677 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:29.380+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:29.677 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:29.380+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:29.677 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:29.381+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:29.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:29.674+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:29.674+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:29.674+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:29.675+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:29.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:29.822+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:29.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:29.822+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:29.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:29.822+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:29.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:29.822+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:30.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:30.391+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:30.391+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:30.391+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:30.392+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:30.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:30.669+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:30.669+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:30.669+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:30.670+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:30.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:30.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:30.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:30.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:30.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:30.856+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:30.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:30.856+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:31.362 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:06:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:06:31.066+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 19079 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:06:31.642 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:31.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:31.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:31.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:31.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:31.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:31.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:31.381+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:31.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:31.638+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:31.638+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:31.638+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:31.638+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:31.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:31.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:31.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:31.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:31.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:31.880+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:32.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:32.363+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:32.363+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:32.363+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:32.364+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:32.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:32.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:32.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:32.609+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:33.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:32.893+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:32.893+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:32.893+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:32.894+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:33.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:33.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:33.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:33.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:33.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:33.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:33.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:33.387+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:33.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:33.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:33.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:33.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:33.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:33.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:33.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:33.564+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:34.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:33.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:34.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:33.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:33.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:33.942+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:34.417+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:34.417+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:34.417+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:34.418+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:34.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:34.591+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:34.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:34.591+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:34.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:34.591+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:34.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:34.592+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:35.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:34.965+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:35.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:34.965+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:35.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:34.965+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:35.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:34.965+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:35.400+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:35.400+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:35.400+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:35.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:35.401+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:35.632+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:35.632+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:35.632+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:35.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:35.633+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:36.376 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:06:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:06:36.067+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 19166 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:06:36.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:35.987+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:36.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:35.987+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:36.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:35.987+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:36.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:35.988+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:36.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:36.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:36.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:36.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:36.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:36.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:36.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:36.416+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:36.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:36.668+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:36.668+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:36.668+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:36.669+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:37.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:36.987+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:37.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:36.987+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:37.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:36.987+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:37.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:36.987+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:37.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:37.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:37.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:37.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:37.385+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:38.031 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:37.717+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:38.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:37.717+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:38.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:37.717+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:38.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:37.717+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:38.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:38.028+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:38.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:38.028+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:38.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:38.028+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:38.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:38.029+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:38.690 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:38.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:38.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:38.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:38.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:38.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:38.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:38.347+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:38.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:38.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:38.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:38.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:38.688+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:39.304 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:39.011+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:39.304 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:39.011+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:39.304 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:39.011+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:39.304 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:39.012+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:39.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:39.300+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:39.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:39.300+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:39.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:39.300+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:39.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:39.301+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:39.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:39.642+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:39.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:39.642+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:39.642+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:39.642+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:40.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:39.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:40.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:39.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:40.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:39.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:40.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:39.966+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:40.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:40.253+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:40.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:40.253+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:40.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:40.253+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:40.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:40.254+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:40.952 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:40.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:40.952 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:40.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:40.952 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:40.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:40.952 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:40.600+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:41.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:06:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:06:41.070+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 19257 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:06:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:40.949+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:40.949+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:40.949+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:40.950+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:41.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:41.228+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:41.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:41.228+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:41.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:41.228+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:41.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:41.228+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:41.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:41.625+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:41.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:41.625+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:41.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:41.625+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:41.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:41.626+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:42.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:41.948+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:42.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:41.948+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:41.948+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:41.949+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:42.576 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:42.229+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:42.576 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:42.229+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:42.576 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:42.229+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:42.576 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:42.230+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:42.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:42.592+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:42.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:42.592+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:42.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:42.592+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:42.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:42.593+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:43.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:42.963+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:43.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:42.963+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:42.963+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:42.964+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:43.594 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:43.227+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:43.594 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:43.227+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:43.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:43.227+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:43.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:43.227+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:43.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:43.625+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:43.625+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:43.625+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:43.625+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:44.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:44.178+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:44.178+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:44.178+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:44.178+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:43.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:43.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:43.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:44.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:43.982+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:44.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:44.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:44.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:44.588+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:45.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:45.184+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:45.184+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:45.184+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:45.185+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:44.974+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:44.974+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:45.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:44.974+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:45.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:44.975+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:45.929 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:45.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:45.930 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:45.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:45.930 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:45.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:45.930 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:45.580+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:06:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:06:46.073+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 19345 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:06:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:46.172+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:46.172+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:46.172+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:46.173+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:45.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:45.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:45.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:45.928+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:46.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:46.600+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:46.600+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:46.600+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:46.601+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:47.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:46.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:47.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:46.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:47.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:46.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:47.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:46.891+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:47.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:47.182+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:47.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:47.182+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:47.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:47.182+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:47.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:47.183+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:47.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:47.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:47.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:47.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:47.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:47.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:47.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:47.579+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:48.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:47.846+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:48.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:47.846+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:48.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:47.846+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:48.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:47.846+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:48.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:48.143+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:48.143+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:48.143+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:48.144+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:48.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:48.629+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:48.629+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:48.629+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:48.629+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:48.807+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:48.807+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:48.807+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:48.808+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:49.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:49.179+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:49.179+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:49.179+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:49.180+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:49.825+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:49.825+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:49.825+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:49.826+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:49.662+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:49.662+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:49.662+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:49.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:49.663+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:50.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:50.203+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:50.203+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:50.203+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:50.203+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:50.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:50.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:50.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:50.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:50.871+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:50.634+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:50.634+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:50.634+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:50.634+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:51.362 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:06:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:06:51.075+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 19437 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:06:51.362 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:51.211+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:51.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:51.211+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:51.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:51.211+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:51.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:51.212+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:51.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:51.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:51.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:51.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:51.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:51.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:51.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:51.644+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:51.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:51.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:51.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:51.917+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:52.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:52.214+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:52.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:52.214+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:52.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:52.214+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:52.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:52.214+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:52.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:52.602+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:52.606 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:52.602+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:52.606 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:52.602+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:52.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:52.602+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:53.204 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:52.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:53.204 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:52.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:53.204 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:52.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:53.204 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:52.931+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:53.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:53.202+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:53.202+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:53.202+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:53.202+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:53.929 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:53.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:53.929 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:53.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:53.930 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:53.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:53.930 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:53.587+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:53.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:53.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:53.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:53.927+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:54.553 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:54.249+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:54.553 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:54.249+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:54.553 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:54.249+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:54.553 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:54.249+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:54.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:54.551+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:54.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:54.551+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:54.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:54.551+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:54.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:54.551+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:55.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:54.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:55.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:54.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:55.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:54.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:55.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:54.889+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:55.570 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:55.290+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:55.571 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:55.290+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:55.571 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:55.290+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:55.571 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:55.291+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:55.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:55.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:55.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:55.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:55.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:55.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:55.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:55.569+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:56.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:06:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:06:56.077+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 19525 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:06:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:55.919+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:55.919+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:55.919+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:55.920+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:56.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:56.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:56.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:56.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:56.332+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:56.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:56.539+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:56.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:56.539+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:56.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:56.539+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:56.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:56.540+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:57.372 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:56.969+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:57.372 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:56.969+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:57.372 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:56.969+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:57.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:56.970+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:57.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:57.370+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:57.370+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:57.370+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:57.371+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:57.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:57.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:57.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:57.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:57.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:57.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:57.588+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:57.925+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:57.925+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:57.925+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:57.926+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:58.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:58.605+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:58.605+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:58.605+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:58.606+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:58.376+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:58.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:58.376+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:58.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:58.376+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:58.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:58.376+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:59.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:58.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:58.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:58.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:06:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:58.910+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:06:59.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:59.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:59.613 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:59.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:59.613 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:59.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:06:59.613 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:06:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:06:59.336+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:06:59.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:59.610+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:59.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:59.610+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:59.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:59.610+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:06:59.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:06:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:06:59.611+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:00.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:59.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:00.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:59.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:00.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:59.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:00.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:06:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:06:59.888+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:00.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:00.312+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:00.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:00.312+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:00.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:00.312+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:00.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:00.313+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:00.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:00.621+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:00.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:00.621+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:00.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:00.621+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:00.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:00.621+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:01.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:07:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:07:01.079+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 19616 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:07:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:00.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:00.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:00.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:00.901+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:01.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:01.314+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:01.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:01.314+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:01.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:01.314+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:01.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:01.314+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:01.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:01.628+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:01.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:01.628+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:01.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:01.628+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:01.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:01.629+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:02.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:01.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:01.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:01.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:01.916+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:02.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:02.302+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:02.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:02.302+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:02.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:02.302+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:02.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:02.303+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:02.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:02.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:02.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:02.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:02.937 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:02.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:02.937 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:02.598+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:02.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:02.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:02.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:02.935+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:03.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:03.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:03.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:03.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:03.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:03.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:03.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:03.316+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:03.596 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:03.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:03.596 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:03.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:03.596 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:03.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:03.597 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:03.555+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:03.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:03.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:03.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:03.910+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:04.590 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:04.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:04.590 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:04.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:04.590 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:04.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:04.590 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:04.357+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:04.911 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:04.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:04.911 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:04.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:04.911 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:04.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:04.911 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:04.588+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:05.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:04.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:05.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:04.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:04.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:04.909+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:05.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:05.361+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:05.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:05.361+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:05.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:05.361+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:05.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:05.362+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:05.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:05.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:05.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:05.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:05.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:05.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:05.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:05.867+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:05.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:05.634+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:05.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:05.634+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:05.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:05.634+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:05.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:05.635+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:06.359 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:07:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:07:06.081+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 19704 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:07:06.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:06.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:06.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:06.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:06.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:06.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:06.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:06.357+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:06.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:06.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:06.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:06.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:06.886+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:06.673+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:06.673+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:06.673+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:06.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:06.674+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:07.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:07.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:07.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:07.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:07.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:07.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:07.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:07.335+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:08.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:07.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:07.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:07.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:07.905+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:07.719+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:07.719+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:07.719+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:07.720+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:08.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:08.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:08.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:08.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:08.336+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:09.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:08.759+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:09.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:08.759+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:09.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:08.759+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:09.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:08.760+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:08.906+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:08.906+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:08.906+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:09.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:08.907+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:09.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:09.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:09.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:09.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:09.295+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:10.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:09.784+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:10.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:09.784+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:10.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:09.784+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:10.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:09.785+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:10.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:09.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:10.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:09.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:10.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:09.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:10.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:09.916+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:10.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:10.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:10.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:10.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:10.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:10.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:10.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:10.294+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:11.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:10.787+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:11.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:10.787+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:11.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:10.787+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:11.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:10.788+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:11.087 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:10.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:11.087 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:10.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:11.087 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:10.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:11.087 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:10.890+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:11.359 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:07:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:07:11.084+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 19796 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:07:11.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:11.256+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:11.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:11.256+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:11.360 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:11.256+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:11.360 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:11.257+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:11.767+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:11.767+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:11.767+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:11.768+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:11.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:12.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:11.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:12.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:11.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:12.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:11.891+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:12.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:12.295+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:12.295+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:12.295+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:12.297+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:13.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:12.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:12.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:12.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:12.880+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:12.728+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:12.728+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:12.728+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:13.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:12.729+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:13.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:13.321+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:13.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:13.321+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:13.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:13.321+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:13.597 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:13.322+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:13.850 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:13.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:13.850 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:13.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:13.850 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:13.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:13.850 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:13.732+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:14.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:13.847+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:13.847+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:13.847+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:13.848+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:14.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:14.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:14.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:14.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:14.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:14.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:14.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:14.349+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:14.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:14.700+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:14.700+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:14.700+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:14.701+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:14.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:14.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:14.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:14.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:14.887+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:15.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:15.318+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:15.318+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:15.318+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:15.318+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:16.088 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:15.733+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:16.089 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:15.733+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:16.089 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:15.733+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:16.089 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:15.734+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:16.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:15.877+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:16.090 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:15.877+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:16.090 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:15.877+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:16.090 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:15.878+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:16.368 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:07:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:07:16.086+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 19884 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:07:16.368 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:16.271+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:16.369 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:16.271+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:16.369 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:16.271+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:16.369 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:16.272+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:16.754+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:16.754+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:16.754+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:16.754+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:16.849+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:16.849+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:17.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:16.849+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:17.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:16.850+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:17.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:17.303+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:17.303+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:17.303+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:17.303+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:18.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:17.797+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:17.797+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:17.797+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:17.798+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:17.860+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:17.860+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:18.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:17.860+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:18.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:17.860+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:18.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:18.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:18.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:18.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:18.345+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:19.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:18.762+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:18.762+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:18.762+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:18.763+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:18.820+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:18.820+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:19.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:18.820+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:19.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:18.820+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:19.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:19.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:19.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:19.345+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:20.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:19.741+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:20.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:19.741+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:20.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:19.741+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:20.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:19.742+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:20.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:19.834+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:20.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:19.834+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:20.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:19.834+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:20.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:19.835+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:20.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:20.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:20.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:20.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:20.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:20.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:20.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:20.392+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:20.705+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:20.705+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:20.705+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:20.706+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:20.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:20.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:20.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:20.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:20.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:20.877+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:21.345 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:07:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:07:21.087+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 19975 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:07:21.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:21.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:21.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:21.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:21.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:21.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:21.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:21.343+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:21.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:21.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:21.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:21.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:21.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:21.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:21.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:21.866+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:21.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:21.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:21.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:21.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:21.711+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:22.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:22.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:22.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:22.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:22.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:22.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:22.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:22.382+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:22.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:22.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:22.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:22.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:22.830+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:22.691+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:22.691+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:22.691+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:22.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:22.692+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:23.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:23.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:23.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:23.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:23.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:23.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:23.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:23.409+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:23.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:23.667+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:23.667+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:23.667+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:23.668+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:23.817+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:23.817+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:23.817+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:23.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:23.818+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:24.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:24.368+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:24.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:24.368+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:24.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:24.368+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:24.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:24.369+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:24.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:24.820+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:24.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:24.820+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:24.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:24.820+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:24.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:24.821+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:24.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:24.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:24.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:24.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:24.664+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:25.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:25.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:25.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:25.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:25.414+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:25.645+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:25.645+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:25.645+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:25.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:25.645+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:26.106 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:07:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:07:26.090+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 20063 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:07:26.106 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:25.853+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:26.106 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:25.853+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:26.106 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:25.853+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:26.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:25.854+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:26.379+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:26.379+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:26.379+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:26.380+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:26.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:26.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:26.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:26.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:26.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:26.597+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:27.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:26.894+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:26.894+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:26.894+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:26.894+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:27.626 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:27.360+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:27.626 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:27.360+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:27.626 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:27.360+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:27.626 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:27.361+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:27.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:27.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:27.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:27.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:27.870+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:27.624+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:27.624+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:27.624+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:27.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:27.624+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:28.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:28.372+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:28.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:28.372+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:28.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:28.372+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:28.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:28.372+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:28.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:28.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:28.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:28.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:28.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:28.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:28.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:28.618+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:29.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:28.848+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:28.848+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:28.848+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:28.849+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:29.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:29.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:29.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:29.331+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:29.571+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:29.571+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:29.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:29.571+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:29.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:29.572+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:30.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:29.892+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:30.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:29.892+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:30.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:29.892+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:30.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:29.893+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:30.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:30.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:30.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:30.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:30.320+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:30.887 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:30.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:30.888 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:30.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:30.888 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:30.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:30.888 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:30.588+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:31.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:07:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:07:31.093+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 20155 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:07:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:30.885+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:30.885+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:30.885+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:30.886+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:31.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:31.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:31.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:31.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:31.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:31.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:31.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:31.276+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:31.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:31.608+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:31.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:31.608+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:31.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:31.608+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:31.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:31.609+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:32.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:31.903+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:31.903+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:31.903+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:31.904+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:32.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:32.305+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:32.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:32.305+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:32.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:32.305+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:32.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:32.306+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:32.916 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:32.575+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:32.916 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:32.575+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:32.916 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:32.575+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:32.916 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:32.576+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:33.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:32.913+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:32.913+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:32.913+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:32.914+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:33.597 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:33.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:33.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:33.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:33.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:33.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:33.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:33.331+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:33.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:33.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:33.599 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:33.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:33.599 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:33.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:33.599 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:33.565+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:33.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:33.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:33.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:33.868+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:34.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:34.287+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:34.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:34.287+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:34.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:34.287+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:34.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:34.288+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:34.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:34.601+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:34.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:34.601+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:34.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:34.601+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:34.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:34.602+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:35.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:34.901+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:34.901+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:34.901+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:34.902+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:35.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:35.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:35.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:35.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:35.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:35.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:35.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:35.285+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:35.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:35.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:35.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:35.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:35.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:35.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:35.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:35.578+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:36.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:07:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:07:36.095+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 20243 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:07:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:35.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:35.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:35.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:35.934+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:36.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:36.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:36.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:36.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:36.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:36.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:36.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:36.315+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:36.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:36.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:36.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:36.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:36.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:36.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:36.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:36.555+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:36.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:36.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:36.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:36.929+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:37.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:37.322+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:37.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:37.322+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:37.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:37.322+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:37.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:37.323+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:37.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:37.546+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:37.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:37.546+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:37.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:37.546+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:37.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:37.547+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:38.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:37.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:37.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:37.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:37.919+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:38.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:38.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:38.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:38.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:38.346+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:38.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:38.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:38.568+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:38.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:38.569+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:39.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:38.962+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:38.962+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:38.962+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:38.962+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:39.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:39.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:39.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:39.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:39.390+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:39.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:39.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:39.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:39.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:39.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:39.599+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:40.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:39.935+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:39.935+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:39.935+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:39.936+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:40.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:40.420+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:40.420+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:40.420+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:40.421+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:40.633+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:40.633+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:40.633+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:40.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:40.633+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:41.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:40.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:41.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:40.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:41.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:40.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:41.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:40.935+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:41.364 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:07:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:07:41.097+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 20334 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:07:41.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:41.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:41.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:41.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:41.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:41.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:41.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:41.401+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:41.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:41.666+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:41.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:41.666+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:41.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:41.666+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:41.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:41.667+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:42.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:41.942+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:41.942+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:41.942+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:41.943+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:42.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:42.618+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:42.618+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:42.618+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:42.618+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:42.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:42.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:42.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:42.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:42.418+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:43.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:42.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:42.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:42.960+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:42.961+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:43.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:43.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:43.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:43.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:43.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:43.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:43.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:43.465+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:43.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:43.582+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:43.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:43.582+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:43.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:43.582+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:43.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:43.583+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:44.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:43.956+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:43.956+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:43.956+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:43.957+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:44.474+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:44.474+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:44.474+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:44.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:44.475+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:44.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:44.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:44.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:44.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:44.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:44.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:44.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:44.609+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:45.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:44.977+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:44.977+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:44.977+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:44.978+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:45.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:45.643+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:45.941 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:45.643+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:45.941 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:45.643+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:45.941 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:45.644+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:45.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:45.508+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:45.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:45.508+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:45.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:45.508+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:45.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:45.509+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:46.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:07:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:07:46.100+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 20419 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:07:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:45.938+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:45.938+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:45.938+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:45.939+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:46.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:46.505+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:46.505+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:46.505+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:46.506+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:46.601+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:46.601+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:46.601+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:46.602+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:46.962+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:46.962+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:46.962+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:46.963+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:47.813 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:47.549+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:47.813 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:47.549+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:47.813 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:47.549+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:47.813 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:47.550+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:47.814 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:47.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:47.814 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:47.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:47.814 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:47.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:47.814 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:47.564+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:47.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:47.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:47.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:47.966+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:48.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:48.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:48.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:48.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:48.609+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:48.526+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:48.526+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:48.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:48.526+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:48.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:48.527+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:48.978+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:48.978+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:48.978+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:48.979+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:49.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:49.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:49.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:49.645+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:49.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:49.527+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:49.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:49.527+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:49.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:49.527+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:49.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:49.528+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:50.021+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:50.021+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:50.021+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:50.022+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:50.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:50.657+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:50.657+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:50.657+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:50.658+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:50.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:50.572+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:50.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:50.572+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:50.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:50.572+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:50.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:50.573+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:51.258 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:07:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:07:51.102+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 20505 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:07:51.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:50.991+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:51.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:50.991+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:51.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:50.991+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:51.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:50.991+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:51.536+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:51.536+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:51.536+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:51.537+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:51.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:51.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:51.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:51.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:51.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:51.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:51.610+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:52.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:52.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:52.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:52.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:52.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:52.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:52.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:52.036+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:52.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:52.640+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:52.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:52.640+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:52.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:52.640+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:52.901 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:52.641+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:52.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:52.524+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:52.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:52.524+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:52.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:52.524+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:52.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:52.525+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:53.054+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:53.054+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:53.054+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:53.055+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:53.552+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:53.552+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:53.552+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:53.553+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:53.659+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:53.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:53.659+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:53.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:53.659+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:53.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:53.660+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:54.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:54.026+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:54.026+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:54.026+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:54.026+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:54.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:54.593+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:54.593+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:54.593+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:54.593+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:54.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:54.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:54.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:54.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:54.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:54.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:54.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:54.694+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:55.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:54.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:54.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:54.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:54.982+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:55.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:55.553+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:55.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:55.553+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:55.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:55.553+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:55.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:55.553+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:55.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:55.719+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:55.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:55.719+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:55.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:55.719+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:55.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:55.720+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:56.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:07:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:07:56.104+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 20587 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:07:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:55.955+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:55.955+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:55.955+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:55.956+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:56.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:56.564+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:56.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:56.564+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:56.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:56.564+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:56.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:56.565+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:56.717+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:56.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:56.717+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:56.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:56.717+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:56.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:56.718+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:57.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:56.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:56.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:56.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:56.979+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:57.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:57.567+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:57.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:57.567+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:57.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:57.567+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:57.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:57.568+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:57.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:57.731+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:57.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:57.731+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:57.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:57.731+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:57.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:57.732+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:57.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:57.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:57.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:57.940+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:58.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:58.592+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:58.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:58.592+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:58.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:58.592+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:58.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:58.593+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:58.895 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:58.683+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:58.895 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:58.683+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:58.895 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:58.683+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:58.895 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:58.683+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:58.892+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:58.892+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:58.892+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:58.893+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:59.681 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:59.575+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:59.681 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:59.575+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:59.681 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:59.575+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:07:59.681 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:07:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:07:59.575+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:07:59.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:59.679+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:59.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:59.679+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:59.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:59.679+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:07:59.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:07:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:07:59.679+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:07:59.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:59.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:59.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:59.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:59.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:59.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:07:59.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:07:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:07:59.901+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:00.872 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:00.562+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:00.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:00.562+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:00.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:00.562+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:00.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:00.562+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:00.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:00.674+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:00.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:00.674+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:00.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:00.674+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:00.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:00.675+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:01.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:08:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:08:01.107+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 20672 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:08:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:00.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:00.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:00.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:00.871+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:01.869 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:01.553+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:01.869 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:01.553+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:01.869 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:01.553+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:01.869 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:01.554+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:01.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:01.720+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:01.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:01.720+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:01.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:01.720+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:01.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:01.721+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:02.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:01.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:01.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:01.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:01.867+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:02.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:02.544+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:02.544+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:02.544+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:02.545+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:03.058 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:02.752+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:03.058 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:02.752+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:03.058 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:02.752+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:03.058 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:02.752+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:03.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:02.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:03.059 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:02.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:03.059 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:02.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:03.059 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:02.895+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:03.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:03.723+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:03.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:03.723+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:03.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:03.723+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:03.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:03.723+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:03.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:03.588+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:03.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:03.588+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:03.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:03.588+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:03.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:03.588+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:03.940+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:03.940+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:03.940+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:03.940+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:04.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:04.608+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:04.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:04.608+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:04.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:04.608+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:04.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:04.608+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:04.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:04.682+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:04.952 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:04.682+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:04.952 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:04.682+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:04.952 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:04.683+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:04.947+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:04.947+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:04.947+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:04.948+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:05.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:05.609+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:05.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:05.609+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:05.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:05.609+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:05.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:05.610+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:05.902 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:05.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:05.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:05.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:05.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:05.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:05.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:05.651+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:06.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:08:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:08:06.109+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 20751 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:08:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:05.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:05.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:05.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:05.900+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:06.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:06.621+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:06.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:06.621+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:06.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:06.621+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:06.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:06.621+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:06.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:06.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:06.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:06.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:06.665+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:06.859+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:06.859+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:06.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:06.859+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:06.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:06.860+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:07.905 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:07.575+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:07.905 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:07.575+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:07.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:07.575+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:07.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:07.576+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:07.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:07.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:07.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:07.618+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:07.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:07.618+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:07.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:07.618+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:08.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:07.902+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:07.902+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:07.902+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:07.903+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:08.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:08.577+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:08.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:08.577+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:08.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:08.577+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:08.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:08.578+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:08.859 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:08.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:08.859 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:08.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:08.859 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:08.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:08.859 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:08.652+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:09.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:08.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:08.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:08.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:08.856+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:09.838 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:09.564+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:09.838 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:09.564+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:09.838 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:09.564+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:09.838 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:09.564+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:09.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:09.655+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:09.839 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:09.655+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:09.839 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:09.655+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:09.839 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:09.655+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:10.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:09.834+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:10.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:09.834+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:10.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:09.834+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:10.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:09.835+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:10.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:10.646+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:10.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:10.646+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:10.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:10.646+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:10.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:10.646+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:10.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:10.526+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:10.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:10.526+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:10.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:10.526+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:10.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:10.527+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:11.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:10.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:11.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:10.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:11.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:10.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:11.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:10.855+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:08:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:08:11.110+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 20838 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:08:11.872 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:11.522+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:11.872 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:11.522+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:11.872 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:11.522+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:11.872 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:11.522+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:11.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:11.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:11.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:11.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:11.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:11.693+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:11.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:11.694+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:12.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:11.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:11.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:11.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:11.870+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:12.830 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:12.485+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:12.830 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:12.485+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:12.830 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:12.485+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:12.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:12.486+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:12.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:12.652+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:12.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:12.652+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:12.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:12.652+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:12.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:12.653+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:13.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:12.827+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:13.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:12.827+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:13.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:12.827+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:13.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:12.828+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:13.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:13.437+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:13.437+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:13.437+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:13.438+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:13.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:13.699+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:13.699+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:13.699+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:13.700+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:13.810+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:13.810+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:13.810+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:13.810+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:14.668+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:14.668+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:14.668+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:14.669+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:14.449+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:14.449+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:14.449+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:14.450+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:15.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:14.852+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:14.852+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:14.852+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:14.852+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:15.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:15.410+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:15.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:15.410+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:15.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:15.410+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:15.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:15.410+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:15.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:15.694+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:15.694+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:15.694+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:15.694+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:15.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:15.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:15.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:15.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:15.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:15.858+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:16.371 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:08:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:08:16.114+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 20923 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:08:16.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:16.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:16.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:16.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:16.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:16.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:16.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:16.388+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:16.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:16.663+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:16.663+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:16.663+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:16.664+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:16.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:16.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:16.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:16.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:16.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:16.844+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:17.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:17.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:17.698 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:17.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:17.698 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:17.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:17.698 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:17.342+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:17.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:17.695+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:17.695+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:17.695+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:17.696+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:17.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:17.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:17.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:17.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:17.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:17.892+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:18.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:18.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:18.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:18.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:18.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:18.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:18.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:18.382+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:18.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:18.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:18.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:18.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:18.921+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:18.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:18.712+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:18.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:18.712+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:18.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:18.712+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:18.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:18.712+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:19.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:19.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:19.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:19.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:19.387+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:20.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:19.728+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:20.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:19.728+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:20.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:19.728+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:20.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:19.730+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:20.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:19.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:20.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:19.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:20.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:19.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:20.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:19.904+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:20.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:20.432+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:20.432+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:20.432+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:20.432+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:21.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:20.727+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:21.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:20.727+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:21.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:20.727+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:21.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:20.728+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:21.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:20.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:21.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:20.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:21.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:20.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:21.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:20.872+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:21.455 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:08:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:08:21.116+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 21011 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:08:21.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:21.453+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:21.453+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:21.453+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:21.454+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:22.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:21.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:21.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:21.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:21.922+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:21.737+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:21.737+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:21.737+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:22.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:21.738+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:22.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:22.465+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:22.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:22.465+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:22.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:22.465+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:22.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:22.466+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:23.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:22.878+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:22.878+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:22.878+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:22.878+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:23.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:22.759+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:23.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:22.759+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:23.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:22.759+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:23.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:22.760+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:23.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:23.487+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:23.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:23.487+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:23.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:23.487+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:23.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:23.487+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:24.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:23.764+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:24.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:23.764+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:24.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:23.764+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:24.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:23.765+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:23.853+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:24.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:23.853+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:24.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:23.853+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:24.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:23.854+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:24.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:24.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:24.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:24.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:24.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:24.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:24.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:24.474+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:25.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:24.751+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:25.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:24.751+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:25.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:24.751+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:25.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:24.752+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:25.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:24.896+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:25.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:24.896+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:25.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:24.896+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:25.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:24.897+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:25.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:25.450+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:25.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:25.450+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:25.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:25.450+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:25.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:25.451+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:25.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:25.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:25.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:25.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:25.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:25.711+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:25.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:25.712+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:25.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:25.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:25.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:25.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:25.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:25.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:25.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:25.933+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:26.375 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:08:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:08:26.118+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 21099 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:08:26.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:26.479+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:26.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:26.479+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:26.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:26.479+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:26.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:26.480+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:26.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:26.703+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:26.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:26.703+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:26.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:26.703+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:26.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:26.704+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:27.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:26.961+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:26.961+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:26.961+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:26.962+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:27.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:27.526+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:27.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:27.526+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:27.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:27.526+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:27.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:27.527+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:27.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:27.691+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:27.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:27.691+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:27.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:27.691+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:27.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:27.691+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:28.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:27.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:27.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:27.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:27.929+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:28.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:28.486+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:28.486+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:28.486+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:28.487+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:28.647+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:28.647+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:28.647+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:28.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:28.648+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:29.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:28.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:28.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:28.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:28.958+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:29.948 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:29.661+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:29.948 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:29.661+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:29.949 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:29.661+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:29.949 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:29.661+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:29.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:29.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:29.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:29.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:29.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:29.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:29.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:29.471+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:30.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:29.946+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:29.946+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:29.946+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:29.947+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:30.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:30.444+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:30.444+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:30.444+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:30.444+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:30.667+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:30.667+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:30.667+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:30.668+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:31.123 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:30.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:31.123 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:30.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:31.123 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:30.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:31.123 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:30.934+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:31.435 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:08:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:08:31.120+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 21191 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:08:31.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:31.433+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:31.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:31.433+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:31.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:31.433+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:31.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:31.434+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:31.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:31.689+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:31.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:31.689+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:31.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:31.689+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:31.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:31.690+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:32.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:31.943+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:31.943+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:31.943+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:31.944+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:32.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:32.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:32.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:32.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:32.418+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:33.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:32.738+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:33.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:32.738+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:33.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:32.738+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:33.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:32.738+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:33.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:32.942+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:33.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:32.942+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:33.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:32.942+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:33.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:32.942+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:33.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:33.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:33.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:33.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:33.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:33.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:33.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:33.387+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:33.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:33.697+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:33.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:33.697+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:33.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:33.697+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:33.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:33.698+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:34.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:33.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:33.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:33.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:33.941+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:34.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:34.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:34.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:34.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:34.417+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:35.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:34.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:34.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:34.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:34.723+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:34.968+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:34.968+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:34.968+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:35.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:34.969+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:35.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:35.441+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:35.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:35.441+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:35.441+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:35.441+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:36.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:35.754+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:36.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:35.754+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:36.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:35.754+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:36.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:35.755+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:36.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:35.986+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:36.128 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:35.986+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:36.128 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:35.986+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:36.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:35.986+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:36.409 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:08:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:08:36.123+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 21279 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:08:36.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:36.407+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:36.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:36.407+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:36.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:36.407+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:36.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:36.407+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:36.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:36.708+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:36.708+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:36.708+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:36.709+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:37.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:37.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:37.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:37.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:37.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:37.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:37.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:37.009+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:37.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:37.360+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:37.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:37.360+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:37.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:37.360+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:37.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:37.361+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:37.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:37.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:37.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:37.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:37.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:37.664+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:37.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:37.665+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:38.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:37.970+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:38.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:37.970+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:38.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:37.970+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:38.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:37.970+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:38.677 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:38.354+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:38.678 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:38.354+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:38.678 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:38.354+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:38.678 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:38.355+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:38.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:38.675+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:38.675+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:38.675+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:38.676+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:39.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:39.001+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:39.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:39.001+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:39.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:39.001+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:39.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:39.002+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:39.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:39.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:39.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:39.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:39.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:39.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:39.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:39.387+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:39.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:39.637+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:39.637+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:39.637+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:39.638+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:40.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:39.965+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:40.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:39.965+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:40.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:39.965+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:40.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:39.966+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:40.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:40.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:40.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:40.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:40.402+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:40.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:40.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:40.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:40.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:40.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:40.613+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:41.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:40.995+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:41.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:40.995+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:41.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:40.995+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:41.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:40.996+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:41.408 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:08:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:08:41.125+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 21370 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:08:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:41.406+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:41.406+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:41.406+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:41.407+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:41.575+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:41.575+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:41.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:41.575+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:41.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:41.576+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:42.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:42.042+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:42.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:42.042+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:42.385 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:42.042+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:42.385 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:42.042+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:42.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:42.454+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:42.454+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:42.454+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:42.455+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:42.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:42.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:42.612+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:42.612+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:43.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:43.011+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:43.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:43.011+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:43.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:43.011+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:43.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:43.011+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:43.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:43.425+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:43.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:43.425+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:43.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:43.425+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:43.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:43.425+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:43.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:43.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:43.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:43.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:43.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:43.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:43.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:43.651+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:44.423 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:43.993+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:44.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:43.993+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:44.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:43.993+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:44.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:43.994+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:44.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:44.421+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:44.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:44.421+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:44.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:44.421+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:44.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:44.421+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:44.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:44.699+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:44.699+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:44.699+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:44.700+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:45.379 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:44.977+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:45.379 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:44.977+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:45.379 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:44.977+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:45.380 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:44.978+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:45.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:45.377+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:45.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:45.377+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:45.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:45.377+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:45.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:45.378+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:45.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:45.679+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:45.679+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:45.679+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:45.680+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:46.332 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:08:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:08:46.128+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 21458 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:08:46.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:45.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:46.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:45.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:46.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:45.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:46.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:45.966+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:46.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:46.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:46.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:46.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:46.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:46.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:46.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:46.330+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:46.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:46.697+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:46.697+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:46.697+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:46.698+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:47.347 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:47.012+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:47.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:47.012+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:47.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:47.012+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:47.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:47.012+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:47.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:47.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:47.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:47.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:47.346+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:47.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:47.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:47.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:47.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:47.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:47.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:47.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:47.715+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:48.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:47.992+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:48.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:47.992+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:48.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:47.992+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:48.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:47.993+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:48.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:48.358+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:48.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:48.358+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:48.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:48.358+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:48.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:48.359+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:48.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:48.670+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:48.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:48.670+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:48.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:48.670+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:48.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:48.671+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:49.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:48.956+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:48.956+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:48.956+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:48.957+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:49.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:49.316+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:49.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:49.316+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:49.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:49.316+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:49.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:49.316+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:49.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:49.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:49.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:49.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:49.650+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:50.338 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:49.978+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:50.338 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:49.978+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:50.338 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:49.978+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:50.338 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:49.978+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:50.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:50.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:50.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:50.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:50.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:50.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:50.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:50.336+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:50.685+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:50.685+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:50.685+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:50.685+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:51.308 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:08:51.130+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 21550 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:08:51.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:51.023+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:51.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:51.023+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:51.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:51.023+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:51.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:51.023+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:51.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:51.306+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:51.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:51.306+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:51.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:51.306+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:51.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:51.307+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:51.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:51.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:51.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:51.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:51.676+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:52.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:51.993+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:52.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:51.993+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:52.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:51.993+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:52.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:51.993+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:52.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:52.278+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:52.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:52.278+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:52.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:52.278+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:52.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:52.279+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:52.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:52.698+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:52.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:52.698+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:52.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:52.698+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:52.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:52.699+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:53.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:52.994+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:53.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:52.994+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:53.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:52.994+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:53.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:52.994+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:53.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:53.274+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:53.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:53.274+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:53.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:53.274+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:53.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:53.275+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:53.862 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:53.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:53.863 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:53.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:53.863 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:53.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:53.863 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:53.652+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:53.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:53.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:53.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:53.959+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:54.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:54.228+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:54.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:54.228+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:54.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:54.228+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:54.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:54.229+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:54.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:54.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:54.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:54.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:54.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:54.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:54.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:54.651+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:55.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:54.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:54.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:54.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:54.929+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:55.620 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:55.234+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:55.620 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:55.234+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:55.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:55.234+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:55.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:55.235+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:55.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:55.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:55.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:55.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:55.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:55.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:55.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:55.618+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:56.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:08:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:08:56.131+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 21639 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:08:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:55.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:55.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:55.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:55.942+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:56.642 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:56.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:56.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:56.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:56.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:56.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:56.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:56.280+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:56.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:56.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:56.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:56.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:56.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:56.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:56.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:56.640+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:57.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:56.952+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:56.952+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:56.952+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:56.952+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:57.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:57.252+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:57.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:57.252+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:57.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:57.252+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:57.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:57.252+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:57.934 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:57.642+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:57.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:57.642+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:57.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:57.642+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:57.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:57.643+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:57.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:57.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:57.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:57.933+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:58.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:58.299+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:58.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:58.299+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:58.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:58.299+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:58.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:58.300+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:58.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:58.633+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:58.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:58.633+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:58.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:58.633+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:58.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:58.634+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:58.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:58.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:58.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:08:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:58.957+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:08:59.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:59.307+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:59.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:59.307+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:59.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:59.307+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:08:59.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:08:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:08:59.308+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:08:59.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:59.670+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:59.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:59.670+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:59.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:59.670+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:08:59.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:08:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:08:59.670+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:00.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:59.968+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:00.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:59.968+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:00.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:59.968+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:00.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:08:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:08:59.969+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:00.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:00.307+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:00.307+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:00.307+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:00.307+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:01.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:00.715+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:01.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:00.715+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:01.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:00.715+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:01.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:00.716+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:01.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:00.938+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:01.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:00.938+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:01.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:00.938+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:01.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:00.939+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:01.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:09:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:09:01.135+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 21730 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:09:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:01.333+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:01.333+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:01.333+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:01.334+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:01.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:01.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:01.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:01.671+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:02.333 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:01.983+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:02.333 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:01.983+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:02.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:01.983+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:02.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:01.984+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:02.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:02.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:02.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:02.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:02.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:02.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:02.683 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:02.332+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:02.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:02.680+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:02.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:02.680+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:02.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:02.680+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:02.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:02.680+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:02.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:02.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:02.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:02.954+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:03.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:03.291+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:03.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:03.291+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:03.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:03.291+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:03.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:03.291+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:03.888 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:03.692+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:03.888 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:03.692+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:03.888 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:03.692+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:03.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:03.693+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:04.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:03.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:03.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:03.989+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:03.989+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:04.653 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:04.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:04.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:04.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:04.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:04.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:04.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:04.312+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:04.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:04.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:04.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:04.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:04.652+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:05.341 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:05.032+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:05.342 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:05.032+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:05.342 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:05.032+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:05.342 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:05.033+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:05.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:05.339+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:05.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:05.339+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:05.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:05.339+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:05.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:05.340+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:05.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:05.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:05.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:05.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:05.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:05.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:05.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:05.650+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:06.373 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:09:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:09:06.137+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 21818 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:09:06.374 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:06.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:06.374 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:06.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:06.374 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:06.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:06.374 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:06.010+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:06.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:06.602+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:06.602+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:06.602+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:06.603+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:06.388+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:06.388+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:06.388+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:06.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:06.388+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:07.385 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:07.054+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:07.385 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:07.054+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:07.386 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:07.054+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:07.386 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:07.055+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:07.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:07.383+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:07.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:07.383+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:07.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:07.383+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:07.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:07.383+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:07.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:07.643+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:07.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:07.643+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:07.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:07.643+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:07.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:07.644+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:08.367 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:08.092+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:08.368 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:08.092+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:08.368 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:08.092+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:08.368 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:08.093+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:08.365+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:08.365+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:08.365+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:08.366+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:08.613+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:08.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:08.613+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:08.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:08.613+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:08.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:08.614+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:09.376 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:09.049+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:09.376 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:09.049+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:09.376 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:09.049+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:09.376 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:09.049+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:09.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:09.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:09.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:09.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:09.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:09.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:09.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:09.374+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:09.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:09.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:09.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:09.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:09.650+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:10.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:10.073+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:10.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:10.073+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:10.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:10.073+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:10.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:10.074+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:10.651 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:10.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:10.651 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:10.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:10.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:10.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:10.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:10.347+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:10.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:10.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:10.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:10.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:10.650+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:09:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:09:11.140+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 21910 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:09:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:11.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:11.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:11.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:11.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:11.329+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:11.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:11.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:11.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:11.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:11.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:11.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:11.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:11.111+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:11.604+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:11.604+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:11.604+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:11.605+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:12.378 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:12.104+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:12.379 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:12.104+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:12.379 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:12.104+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:12.379 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:12.105+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:12.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:12.376+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:12.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:12.376+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:12.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:12.376+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:12.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:12.377+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:12.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:12.641+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:12.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:12.641+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:12.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:12.641+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:12.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:12.642+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:13.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:13.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:13.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:13.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:13.331+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:13.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:13.118+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:13.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:13.118+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:13.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:13.118+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:13.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:13.118+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:13.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:13.604+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:13.604+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:13.604+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:13.605+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:14.322 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:14.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:14.322 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:14.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:14.322 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:14.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:14.323 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:14.069+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:14.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:14.320+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:14.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:14.320+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:14.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:14.320+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:14.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:14.320+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:14.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:14.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:14.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:14.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:14.621+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:15.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:15.337+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:15.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:15.337+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:15.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:15.337+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:15.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:15.337+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:15.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:15.103+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:15.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:15.103+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:15.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:15.103+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:15.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:15.104+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:15.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:15.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:15.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:15.661+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:16.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:09:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:09:16.142+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 21998 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:09:16.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:16.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:16.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:16.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:16.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:16.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:16.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:16.356+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:16.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:16.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:16.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:16.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:16.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:16.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:16.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:16.132+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:16.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:16.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:16.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:16.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:16.677+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:17.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:17.350+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:17.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:17.350+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:17.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:17.350+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:17.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:17.351+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:17.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:17.101+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:17.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:17.101+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:17.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:17.101+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:17.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:17.102+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:18.116 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:17.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:18.116 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:17.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:18.116 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:17.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:18.117 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:17.715+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:18.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:18.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:18.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:18.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:18.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:18.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:18.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:18.114+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:18.323+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:18.323+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:18.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:18.323+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:18.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:18.324+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:19.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:18.738+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:19.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:18.738+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:19.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:18.738+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:19.097 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:18.739+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:19.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:19.094+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:19.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:19.094+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:19.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:19.094+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:19.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:19.095+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:19.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:19.336+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:19.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:19.336+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:19.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:19.336+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:19.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:19.337+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:20.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:19.717+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:20.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:19.717+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:20.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:19.717+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:20.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:19.718+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:20.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:20.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:20.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:20.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:20.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:20.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:20.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:20.077+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:20.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:20.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:20.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:20.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:20.343+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:21.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:20.761+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:21.043 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:20.761+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:21.043 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:20.761+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:21.043 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:20.762+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:21.362 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:09:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:09:21.145+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 22089 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:09:21.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:21.040+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:21.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:21.040+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:21.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:21.040+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:21.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:21.041+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:21.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:21.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:21.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:21.374+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:22.066 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:21.801+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:22.066 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:21.801+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:22.066 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:21.801+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:22.067 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:21.802+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:22.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:22.064+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:22.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:22.064+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:22.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:22.064+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:22.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:22.064+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:22.336+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:22.336+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:22.336+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:22.337+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:23.092 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:22.771+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:23.093 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:22.771+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:23.093 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:22.771+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:23.093 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:22.771+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:23.090+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:23.090+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:23.090+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:23.091+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:23.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:23.333+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:23.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:23.333+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:23.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:23.333+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:23.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:23.334+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:23.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:23.782+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:23.782+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:23.782+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:23.783+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:24.125+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:24.125+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:24.125+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:24.126+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:24.364+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:24.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:24.364+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:24.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:24.364+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:24.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:24.365+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:25.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:24.775+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:25.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:24.775+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:25.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:24.775+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:25.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:24.775+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:25.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:25.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:25.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:25.342+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:25.154+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:25.154+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:25.154+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:25.155+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:26.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:25.804+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:26.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:25.804+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:26.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:25.804+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:26.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:25.805+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:26.374 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:09:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:09:26.147+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 22177 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:09:26.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:26.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:26.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:26.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:26.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:26.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:26.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:26.316+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:26.375 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:26.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:26.375 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:26.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:26.375 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:26.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:26.375 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:26.111+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:27.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:26.811+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:27.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:26.811+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:27.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:26.811+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:27.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:26.812+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:27.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:27.129+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:27.129+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:27.129+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:27.130+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:27.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:27.310+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:27.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:27.310+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:27.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:27.310+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:27.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:27.311+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:28.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:27.765+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:28.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:27.765+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:28.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:27.765+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:28.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:27.766+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:28.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:28.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:28.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:28.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:28.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:28.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:28.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:28.082+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:28.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:28.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:28.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:28.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:28.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:28.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:28.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:28.277+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:29.060 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:28.806+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:29.061 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:28.806+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:29.061 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:28.806+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:29.061 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:28.807+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:29.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:29.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:29.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:29.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:29.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:29.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:29.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:29.059+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:29.439 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:29.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:29.439 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:29.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:29.439 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:29.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:29.439 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:29.299+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:30.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:29.852+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:30.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:29.852+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:30.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:29.852+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:30.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:29.853+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:30.189 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:30.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:30.189 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:30.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:30.189 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:30.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:30.189 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:30.058+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:30.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:30.318+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:30.318+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:30.318+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:30.318+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:31.068 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:30.860+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:31.068 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:30.860+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:31.068 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:30.860+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:31.069 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:30.861+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:31.358 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:09:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:09:31.150+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 22269 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:09:31.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:31.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:31.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:31.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:31.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:31.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:31.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:31.298+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:31.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:31.065+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:31.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:31.065+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:31.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:31.065+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:31.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:31.066+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:32.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:32.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:32.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:32.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:32.059+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:31.895+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:31.895+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:32.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:31.895+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:32.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:31.896+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:32.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:32.283+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:32.283+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:32.283+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:32.283+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:32.856+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:32.856+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:32.856+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:32.857+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:33.041+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:33.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:33.041+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:33.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:33.041+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:33.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:33.041+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:33.606 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:33.274+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:33.606 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:33.274+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:33.606 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:33.274+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:33.606 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:33.274+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:34.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:33.869+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:34.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:33.869+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:34.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:33.869+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:34.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:33.869+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:34.011+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:34.011+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:34.011+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:34.012+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:34.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:34.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:34.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:34.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:34.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:34.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:34.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:34.238+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:34.833+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:34.833+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:34.833+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:35.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:34.834+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:35.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:35.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:35.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:35.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:35.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:35.037+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:35.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:35.288+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:35.288+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:35.288+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:35.288+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:36.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:35.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:36.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:35.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:36.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:35.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:36.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:35.833+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:36.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:36.048+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:36.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:36.048+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:36.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:36.048+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:36.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:36.048+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:36.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:09:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:09:36.152+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 22357 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:09:36.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:36.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:36.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:36.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:36.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:36.331+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:36.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:36.331+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:36.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:36.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:36.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:36.823+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:37.035+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:37.035+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:37.035+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:37.036+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:37.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:37.296+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:37.296+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:37.296+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:37.297+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:38.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:37.780+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:38.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:37.780+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:38.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:37.780+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:38.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:37.781+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:38.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:38.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:38.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:38.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:38.298+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:38.083+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:38.083+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:38.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:38.083+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:38.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:38.084+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:39.046 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:38.748+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:39.046 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:38.748+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:39.046 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:38.748+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:39.047 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:38.749+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:39.342 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:39.043+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:39.343 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:39.043+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:39.343 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:39.043+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:39.343 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:39.044+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:39.593 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:39.339+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:39.594 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:39.339+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:39.594 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:39.339+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:39.594 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:39.340+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:39.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:39.787+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:39.787+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:39.787+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:39.787+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:40.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:40.088+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:40.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:40.088+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:40.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:40.088+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:40.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:40.089+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:40.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:40.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:40.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:40.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:40.388+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:41.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:40.783+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:41.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:40.783+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:41.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:40.783+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:41.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:40.784+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:41.373 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:09:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:09:41.154+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 22448 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:09:41.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:41.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:41.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:41.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:41.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:41.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:41.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:41.109+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:41.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:41.369+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:41.369+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:41.369+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:41.370+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:42.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:41.736+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:42.135 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:41.736+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:42.135 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:41.736+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:42.135 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:41.736+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:42.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:42.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:42.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:42.330+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:42.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:42.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:42.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:42.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:42.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:42.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:42.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:42.132+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:43.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:42.784+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:43.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:42.784+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:43.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:42.784+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:43.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:42.784+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:43.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:43.327+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:43.327+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:43.327+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:43.327+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:43.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:43.169+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:43.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:43.169+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:43.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:43.169+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:43.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:43.169+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:43.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:43.784+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:43.784+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:43.784+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:43.784+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:44.338+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:44.338+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:44.338+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:44.339+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:44.165+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:44.165+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:44.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:44.165+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:44.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:44.166+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:44.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:44.797+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:44.797+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:44.797+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:44.798+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:45.124+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:45.124+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:45.124+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:45.125+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:45.322+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:45.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:45.322+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:45.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:45.322+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:45.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:45.322+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:46.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:45.791+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:46.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:45.791+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:46.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:45.791+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:46.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:45.791+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:46.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:09:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:09:46.157+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 22536 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:09:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:46.354+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:46.354+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:46.354+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:46.354+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:46.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:46.160+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:46.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:46.160+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:46.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:46.160+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:46.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:46.161+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:47.196 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:46.820+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:47.196 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:46.820+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:47.196 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:46.820+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:47.196 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:46.821+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:47.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:47.193+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:47.193+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:47.193+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:47.194+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:47.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:47.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:47.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:47.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:47.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:47.349+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:48.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:47.839+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:48.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:47.839+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:48.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:47.839+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:48.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:47.840+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:48.189+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:48.189+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:48.189+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:48.190+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:48.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:48.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:48.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:48.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:48.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:48.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:48.393+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:48.880+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:48.880+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:48.880+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:48.880+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:49.225+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:49.225+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:49.225+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:49.226+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:49.402+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:49.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:49.402+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:49.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:49.402+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:49.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:49.403+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:50.189 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:49.918+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:50.189 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:49.918+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:50.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:49.918+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:50.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:49.919+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:50.360+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:50.360+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:50.360+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:50.361+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:50.247+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:50.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:50.247+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:50.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:50.247+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:50.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:50.247+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:51.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:50.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:51.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:50.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:51.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:50.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:51.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:50.878+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:51.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:09:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:09:51.160+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 22628 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:09:51.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:51.292+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:51.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:51.292+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:51.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:51.292+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:51.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:51.293+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:51.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:51.383+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:51.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:51.383+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:51.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:51.383+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:51.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:51.384+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:51.922+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:51.922+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:51.922+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:51.923+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:52.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:52.264+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:52.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:52.264+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:52.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:52.264+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:52.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:52.264+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:52.410+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:52.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:52.410+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:52.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:52.410+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:52.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:52.411+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:53.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:52.888+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:52.888+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:52.888+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:52.889+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:53.608 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:53.450+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:53.608 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:53.450+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:53.608 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:53.450+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:53.609 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:53.451+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:53.609 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:53.285+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:53.609 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:53.285+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:53.609 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:53.285+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:53.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:53.285+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:54.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:53.916+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:53.916+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:53.916+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:53.917+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:54.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:54.472+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:54.472+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:54.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:54.473+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:54.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:54.258+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:54.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:54.258+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:54.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:54.258+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:54.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:54.258+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:54.881+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:54.881+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:54.881+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:54.882+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:55.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:55.235+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:55.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:55.235+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:55.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:55.235+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:55.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:55.236+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:55.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:55.497+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:55.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:55.497+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:55.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:55.497+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:55.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:55.498+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:56.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:55.865+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:56.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:55.865+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:56.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:55.865+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:56.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:55.866+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:56.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:09:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:09:56.162+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 22716 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:09:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:56.237+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:56.237+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:56.237+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:56.237+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:56.890 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:56.509+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:56.890 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:56.509+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:56.890 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:56.509+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:56.890 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:56.509+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:57.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:56.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:56.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:56.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:56.888+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:57.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:57.249+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:57.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:57.249+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:57.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:57.249+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:57.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:57.250+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:57.860 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:57.555+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:57.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:57.555+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:57.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:57.555+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:57.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:57.555+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:57.926+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:57.926+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:57.926+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:57.926+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:58.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:58.222+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:58.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:58.222+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:58.222+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:58.222+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:58.905 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:58.517+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:58.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:58.517+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:58.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:58.517+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:58.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:58.518+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:59.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:58.903+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:58.903+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:58.903+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:09:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:58.904+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:09:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:59.481+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:59.481+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:59.481+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:09:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:09:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:09:59.482+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:09:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:59.238+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:59.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:59.238+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:59.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:59.238+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:09:59.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:09:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:09:59.239+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:00.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:59.899+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:00.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:59.899+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:00.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:59.899+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:00.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:09:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:09:59.899+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:00.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:00.271+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:00.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:00.271+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:00.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:00.271+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:00.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:00.271+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:00.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:00.466+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:00.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:00.466+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:00.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:00.466+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:00.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:00.467+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:01.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:00.945+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:01.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:00.945+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:01.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:00.945+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:01.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:00.945+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:01.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:10:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:10:01.165+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 22807 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:10:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:01.226+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:01.226+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:01.226+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:01.227+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:01.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:01.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:01.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:01.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:01.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:01.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:01.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:01.465+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:01.948+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:01.948+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:01.948+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:01.949+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:02.197+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:02.197+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:02.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:02.197+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:02.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:02.198+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:02.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:02.478+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:02.478+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:02.478+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:02.479+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:03.435 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:02.975+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:03.435 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:02.975+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:03.435 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:02.975+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:03.435 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:02.976+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:03.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:03.187+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:03.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:03.187+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:03.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:03.187+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:03.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:03.188+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:03.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:03.432+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:03.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:03.432+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:03.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:03.432+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:03.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:03.433+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:04.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:03.952+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:04.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:03.952+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:04.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:03.952+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:04.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:03.953+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:04.449 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:04.211+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:04.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:04.211+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:04.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:04.211+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:04.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:04.212+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:04.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:04.447+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:04.447+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:04.447+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:04.448+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:04.942+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:04.942+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:04.942+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:04.943+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:05.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:05.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:05.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:05.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:05.417+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:05.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:05.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:05.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:05.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:05.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:05.233+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:06.170 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:05.914+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:06.170 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:05.914+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:06.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:05.914+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:06.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:05.915+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:06.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:10:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:10:06.167+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 22895 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:10:06.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:06.213+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:06.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:06.213+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:06.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:06.213+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:06.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:06.214+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:06.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:06.466+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:06.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:06.466+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:06.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:06.466+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:06.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:06.467+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:07.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:06.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:07.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:06.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:07.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:06.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:07.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:06.891+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:07.204+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:07.204+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:07.204+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:07.205+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:07.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:07.494+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:07.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:07.494+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:07.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:07.494+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:07.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:07.494+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:08.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:07.913+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:07.913+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:07.913+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:07.914+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:08.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:08.228+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:08.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:08.228+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:08.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:08.228+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:08.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:08.228+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:08.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:08.530+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:08.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:08.530+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:08.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:08.530+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:08.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:08.531+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:09.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:08.921+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:09.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:08.921+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:09.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:08.921+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:09.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:08.921+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:09.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:09.276+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:09.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:09.276+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:09.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:09.276+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:09.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:09.277+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:09.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:09.491+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:09.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:09.491+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:09.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:09.491+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:09.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:09.491+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:10.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:09.962+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:09.962+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:09.962+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:09.963+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:10.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:10.489+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:10.489+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:10.489+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:10.490+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:10.254+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:10.254+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:10.254+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:10.255+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:11.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:10.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:11.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:10.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:11.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:10.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:11.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:10.935+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:11.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:10:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:10:11.170+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 22987 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:10:11.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:11.290+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:11.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:11.290+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:11.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:11.290+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:11.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:11.290+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:11.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:11.497+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:11.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:11.497+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:11.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:11.497+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:11.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:11.497+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:12.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:11.914+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:11.914+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:11.914+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:11.915+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:12.511+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:12.511+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:12.511+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:12.512+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:12.302+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:12.302+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:12.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:12.302+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:12.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:12.303+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:12.875+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:12.875+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:12.875+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:12.875+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:13.609 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:13.510+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:13.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:13.510+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:13.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:13.510+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:13.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:13.511+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:13.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:13.343+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:13.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:13.343+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:13.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:13.343+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:13.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:13.343+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:13.842+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:13.842+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:13.842+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:13.843+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:14.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:14.370+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:14.370+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:14.370+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:14.371+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:14.547+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:14.547+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:14.547+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:14.548+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:15.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:14.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:15.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:14.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:15.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:14.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:15.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:14.848+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:15.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:15.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:15.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:15.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:15.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:15.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:15.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:15.353+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:15.897 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:15.587+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:15.897 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:15.587+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:15.897 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:15.587+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:15.897 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:15.587+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:16.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:15.894+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:16.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:15.894+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:16.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:15.894+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:16.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:15.895+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:16.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:10:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:10:16.173+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 23075 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:10:16.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:16.316+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:16.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:16.316+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:16.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:16.316+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:16.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:16.317+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:16.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:16.585+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:16.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:16.585+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:16.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:16.585+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:16.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:16.586+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:17.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:16.879+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:16.879+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:16.879+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:16.879+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:17.577+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:17.577+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:17.577+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:17.577+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:17.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:17.353+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:17.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:17.353+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:17.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:17.353+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:17.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:17.354+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:18.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:17.839+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:17.839+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:17.840+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:17.840+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:18.531+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:18.531+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:18.531+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:18.532+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:18.386+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:18.386+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:18.386+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:18.386+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:19.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:18.842+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:18.842+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:18.842+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:18.843+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:19.374+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:19.374+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:19.374+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:19.375+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:19.484+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:19.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:19.484+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:19.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:19.484+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:19.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:19.485+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:20.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:19.815+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:20.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:19.815+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:20.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:19.815+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:20.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:19.816+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:20.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:20.326+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:20.326+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:20.326+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:20.326+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:20.822 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:20.506+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:20.822 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:20.506+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:20.822 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:20.506+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:20.822 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:20.507+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:21.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:20.819+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:21.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:20.819+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:21.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:20.819+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:21.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:20.820+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:21.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:10:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:10:21.175+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 23166 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:10:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:21.374+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:21.374+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:21.374+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:21.375+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:21.789 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:21.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:21.789 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:21.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:21.789 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:21.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:21.789 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:21.472+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:21.786+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:21.786+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:21.786+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:21.787+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:22.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:22.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:22.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:22.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:22.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:22.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:22.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:22.337+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:22.496+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:22.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:22.496+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:22.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:22.496+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:22.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:22.497+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:23.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:22.779+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:23.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:22.779+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:23.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:22.779+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:23.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:22.780+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:23.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:23.505+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:23.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:23.505+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:23.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:23.505+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:23.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:23.506+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:23.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:23.304+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:23.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:23.304+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:23.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:23.304+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:23.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:23.305+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:23.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:23.810+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:23.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:23.810+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:23.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:23.810+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:23.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:23.811+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:23.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T20:10:23.941+0000 7f40bd04c640 0 monclient(hunting): authenticate timed out after 300 2024-09-20T20:10:23.945 INFO:teuthology.orchestra.run.smithi114.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-09-20T20:10:24.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:24.333+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:24.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:24.333+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:24.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:24.333+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:24.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:24.334+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:24.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:24.460+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:24.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:24.460+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:24.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:24.460+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:24.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:24.461+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:24.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T20:10:24.712 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-09-20T20:10:24.712 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T20:10:24.712 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-20T20:10:24.730 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-20T20:10:24.730 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-20T20:10:24.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-20T20:10:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:24.836+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:24.836+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:24.836+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:24.837+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:25.088 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T20:10:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:25.446+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:25.446+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:25.446+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:25.447+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:25.314+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:25.314+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:25.314+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:25.315+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:26.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:25.805+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:26.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:25.805+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:26.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:25.805+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:26.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:25.806+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:26.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:10:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:10:26.177+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 23254 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:10:26.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:26.330+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:26.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:26.330+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:26.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:26.330+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:26.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:26.331+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:26.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:26.488+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:26.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:26.488+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:26.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:26.488+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:26.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:26.489+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:27.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:26.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:26.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:26.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:26.833+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:27.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:27.361+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:27.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:27.361+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:27.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:27.361+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:27.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:27.362+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:27.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:27.532+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:27.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:27.532+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:27.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:27.532+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:27.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:27.533+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:28.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:27.814+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:27.814+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:27.814+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:27.815+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:28.489+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:28.489+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:28.489+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:28.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:28.490+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:28.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:28.339+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:28.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:28.339+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:28.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:28.339+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:28.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:28.339+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:29.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:28.795+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:29.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:28.795+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:29.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:28.795+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:29.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:28.796+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:29.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:29.532+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:29.532+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:29.532+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:29.533+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:29.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:29.366+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:29.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:29.366+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:29.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:29.366+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:29.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:29.367+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:30.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:29.761+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:30.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:29.761+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:30.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:29.761+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:30.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:29.761+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:30.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:30.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:30.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:30.359+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:30.548+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:30.548+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:30.548+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:30.548+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:30.735+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:30.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:30.735+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:30.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:30.735+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:30.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:30.736+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:31.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:10:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:10:31.180+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 23346 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:10:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:31.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:31.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:31.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:31.352+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:31.583+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:31.583+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:31.583+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:31.584+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:31.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:31.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:31.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:31.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:31.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:31.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:31.722+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:32.626 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:32.356+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:32.626 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:32.356+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:32.626 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:32.356+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:32.626 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:32.357+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:32.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:32.719+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:32.719+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:32.719+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:32.719+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:32.622+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:32.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:32.622+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:32.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:32.622+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:32.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:32.623+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:33.663 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:33.625+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:33.663 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:33.625+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:33.664 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:33.625+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:33.664 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:33.626+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:33.664 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:33.390+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:33.664 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:33.390+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:33.664 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:33.390+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:33.664 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:33.390+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:33.684+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:33.684+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:33.684+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:33.685+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:34.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:34.633+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:34.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:34.633+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:34.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:34.633+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:34.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:34.634+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:34.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:34.404+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:34.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:34.404+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:34.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:34.404+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:34.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:34.405+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:34.671 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:34.645+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:34.671 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:34.645+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:34.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:34.645+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:34.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:34.646+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:35.698 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:35.593+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:35.698 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:35.593+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:35.698 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:35.593+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:35.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:35.594+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:35.699 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:35.441+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:35.699 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:35.441+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:35.699 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:35.441+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:35.699 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:35.441+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:35.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:35.695+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:35.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:35.695+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:35.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:35.695+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:35.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:35.696+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:36.368 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:10:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:10:36.183+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 23434 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:10:36.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:36.590+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:36.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:36.590+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:36.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:36.590+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:36.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:36.590+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:36.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:36.461+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:36.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:36.461+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:36.706 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:36.461+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:36.706 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:36.462+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:36.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:36.702+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:36.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:36.702+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:36.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:36.702+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:36.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:36.703+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:37.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:37.449+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:37.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:37.449+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:37.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:37.449+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:37.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:37.450+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:37.553+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:37.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:37.553+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:37.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:37.553+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:37.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:37.554+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:37.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:37.690+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:37.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:37.690+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:37.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:37.690+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:37.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:37.691+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:38.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:38.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:38.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:38.687+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:38.553+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:38.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:38.553+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:38.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:38.553+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:38.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:38.554+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:38.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:38.481+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:38.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:38.481+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:38.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:38.481+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:38.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:38.481+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:39.599+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:39.599+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:39.599+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:39.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:39.599+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:39.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:39.432+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:39.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:39.432+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:39.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:39.432+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:39.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:39.433+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:40.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:39.720+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:40.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:39.720+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:40.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:39.720+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:40.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:39.720+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:40.420+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:40.420+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:40.420+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:40.420+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:40.600+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:40.600+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:40.600+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:40.601+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:40.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:40.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:40.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:40.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:40.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:40.714+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:40.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:40.715+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:41.455 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:10:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:10:41.186+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 23525 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:10:41.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:41.558+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:41.558+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:41.558+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:41.559+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:41.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:41.453+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:41.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:41.453+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:41.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:41.453+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:41.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:41.454+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:42.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:41.726+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:42.194 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:41.726+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:42.194 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:41.726+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:42.194 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:41.726+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:42.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:42.707+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:42.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:42.707+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:42.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:42.707+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:42.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:42.708+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:42.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:42.555+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:42.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:42.555+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:42.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:42.555+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:42.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:42.556+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:42.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:42.494+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:42.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:42.494+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:42.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:42.494+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:42.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:42.494+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:43.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:43.560+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:43.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:43.560+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:43.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:43.560+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:43.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:43.561+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:43.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:43.683+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:43.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:43.683+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:43.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:43.683+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:43.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:43.684+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:43.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:43.449+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:43.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:43.449+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:43.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:43.449+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:43.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:43.450+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:44.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:44.519+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:44.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:44.519+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:44.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:44.519+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:44.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:44.520+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:44.646+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:44.646+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:44.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:44.646+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:44.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:44.646+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:44.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:44.456+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:44.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:44.456+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:44.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:44.456+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:44.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:44.457+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:45.514 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:45.512+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:45.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:45.512+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:45.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:45.512+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:45.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:45.457+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:45.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:45.457+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:45.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:45.457+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:45.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:45.458+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:45.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:45.513+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:45.659+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:45.659+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:45.659+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:45.659+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:46.454 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:10:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:10:46.188+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 23613 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:10:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:46.539+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:46.539+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:46.539+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:46.539+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:46.662+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:46.662+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:46.662+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:46.663+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:46.452+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:46.452+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:46.452+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:46.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:46.453+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:47.491+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:47.491+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:47.491+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:47.492+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:47.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:47.692+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:47.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:47.692+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:47.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:47.692+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:47.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:47.693+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:47.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:47.450+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:47.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:47.450+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:47.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:47.450+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:47.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:47.451+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:48.508+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:48.508+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:48.508+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:48.508+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:48.710+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:48.710+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:48.710+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:48.711+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:48.494+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:48.494+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:48.494+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:48.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:48.495+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:49.517+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:49.517+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:49.517+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:49.518+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:49.740+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:49.740+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:49.740+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:49.741+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:49.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:49.521+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:49.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:49.521+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:49.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:49.521+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:49.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:49.522+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:50.536+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:50.536+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:50.536+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:50.536+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:50.718+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:50.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:50.718+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:50.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:50.718+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:50.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:50.718+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:50.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:50.515+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:50.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:50.515+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:50.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:50.515+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:50.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:50.515+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:51.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:10:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:10:51.191+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 23705 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:10:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:51.474+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:51.474+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:51.474+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:51.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:51.475+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:51.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:51.701+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:51.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:51.701+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:51.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:51.701+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:51.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:51.702+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:51.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:51.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:51.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:51.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:51.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:51.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:51.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:51.562+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:52.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:52.478+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:52.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:52.478+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:52.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:52.478+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:52.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:52.479+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:52.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:52.594+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:52.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:52.594+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:52.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:52.594+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:52.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:52.595+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:52.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:52.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:52.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:52.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:52.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:52.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:52.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:52.661+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:53.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:53.461+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:53.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:53.461+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:53.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:53.461+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:53.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:53.461+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:53.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:53.566+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:53.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:53.566+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:53.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:53.566+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:53.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:53.567+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:53.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:53.624+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:53.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:53.624+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:53.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:53.624+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:53.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:53.624+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:54.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:54.602+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:54.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:54.602+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:54.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:54.602+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:54.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:54.603+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:54.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:54.465+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:54.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:54.465+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:54.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:54.465+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:54.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:54.466+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:54.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:54.589+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:54.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:54.589+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:54.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:54.589+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:54.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:54.590+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:55.785 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:55.588+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:55.785 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:55.588+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:55.785 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:55.588+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:55.785 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:55.588+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:55.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:55.491+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:55.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:55.491+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:55.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:55.491+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:55.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:55.491+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:55.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:55.584+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:55.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:55.584+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:55.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:55.584+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:55.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:55.585+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:56.463 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:10:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:10:56.194+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 23793 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:10:56.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:56.560+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:56.560+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:56.560+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:56.560+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:56.460+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:56.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:56.460+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:56.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:56.460+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:56.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:56.461+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:56.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:56.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:56.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:56.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:56.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:56.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:56.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:56.579+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:57.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:57.545+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:57.545+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:57.545+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:57.546+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:57.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:57.595+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:57.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:57.595+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:57.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:57.595+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:57.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:57.596+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:57.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:57.430+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:57.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:57.430+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:57.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:57.430+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:57.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:57.431+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:58.551+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:58.551+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:58.551+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:58.551+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:58.479+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:58.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:58.479+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:58.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:58.479+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:58.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:58.480+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:58.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:58.509+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:58.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:58.509+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:58.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:58.509+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:58.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:58.509+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:59.519+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:59.520+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:59.520+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:10:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:10:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:10:59.520+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:10:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:59.443+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:59.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:59.443+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:59.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:59.443+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:10:59.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:10:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:10:59.443+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:10:59.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:59.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:59.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:59.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:59.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:59.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:10:59.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:10:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:10:59.505+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:00.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:00.485+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:00.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:00.485+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:00.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:00.485+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:00.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:00.485+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:00.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:00.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:00.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:00.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:00.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:00.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:00.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:00.523+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:00.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:00.477+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:00.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:00.477+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:00.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:00.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:00.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:00.478+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:01.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:11:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:11:01.197+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 23918 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:11:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:01.434+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:01.434+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:01.434+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:01.434+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:01.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:01.499+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:01.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:01.499+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:01.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:01.499+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:01.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:01.500+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:01.494+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:01.494+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:01.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:01.494+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:01.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:01.495+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:02.472+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:02.472+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:02.472+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:02.473+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:02.504+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:02.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:02.504+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:02.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:02.504+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:02.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:02.505+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:02.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:02.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:02.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:02.411+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:03.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:03.502+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:03.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:03.502+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:03.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:03.502+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:03.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:03.502+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:03.698 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:03.507+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:03.698 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:03.507+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:03.699 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:03.507+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:03.699 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:03.508+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:03.699 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:03.420+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:03.699 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:03.420+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:03.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:03.420+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:03.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:03.421+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:04.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:04.406+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:04.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:04.406+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:04.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:04.406+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:04.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:04.407+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:04.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:04.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:04.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:04.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:04.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:04.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:04.551+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:04.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:04.459+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:04.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:04.459+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:04.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:04.459+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:04.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:04.460+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:05.447+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:05.447+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:05.447+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:05.448+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:05.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:05.540+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:05.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:05.540+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:05.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:05.540+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:05.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:05.541+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:05.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:05.362+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:05.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:05.362+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:05.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:05.362+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:05.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:05.363+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:06.452 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:11:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:11:06.199+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 24006 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:11:06.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:06.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:06.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:06.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:06.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:06.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:06.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:06.336+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:06.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:06.553+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:06.553+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:06.553+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:06.553+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:06.450+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:06.450+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:06.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:06.450+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:06.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:06.451+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:07.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:07.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:07.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:07.542+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:07.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:07.325+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:07.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:07.325+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:07.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:07.325+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:07.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:07.326+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:07.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:07.434+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:07.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:07.434+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:07.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:07.434+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:07.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:07.435+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:08.480+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:08.480+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:08.480+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:08.481+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:08.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:08.584+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:08.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:08.584+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:08.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:08.584+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:08.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:08.584+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:08.369+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:08.369+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:08.369+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:08.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:08.369+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:09.635 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:09.353+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:09.635 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:09.353+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:09.635 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:09.353+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:09.635 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:09.354+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:09.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:09.460+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:09.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:09.460+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:09.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:09.460+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:09.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:09.461+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:09.632+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:09.632+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:09.632+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:09.633+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:10.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:10.398+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:10.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:10.398+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:10.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:10.398+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:10.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:10.398+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:10.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:10.490+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:10.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:10.490+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:10.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:10.490+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:10.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:10.491+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:10.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:10.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:10.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:10.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:10.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:10.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:10.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:10.644+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:11.361 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:11:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:11:11.201+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 24098 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:11:11.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:11.422+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:11.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:11.422+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:11.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:11.422+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:11.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:11.423+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:11.642 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:11.526+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:11.642 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:11.526+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:11.642 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:11.526+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:11.642 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:11.527+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:11.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:11.638+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:11.638+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:11.638+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:11.639+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:12.632+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:12.632+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:12.632+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:12.633+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:12.487+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:12.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:12.487+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:12.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:12.487+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:12.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:12.487+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:12.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:12.438+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:12.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:12.438+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:12.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:12.438+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:12.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:12.439+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:13.475+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:13.475+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:13.475+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:13.475+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:13.621+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:13.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:13.621+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:13.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:13.621+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:13.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:13.621+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:13.489+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:13.489+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:13.489+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:13.489+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:14.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:14.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:14.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:14.617+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:14.618+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:14.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:14.495+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:14.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:14.495+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:14.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:14.495+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:14.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:14.496+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:14.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:14.510+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:14.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:14.510+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:14.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:14.510+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:14.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:14.511+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:15.517+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:15.517+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:15.517+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:15.518+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:15.462+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:15.462+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:15.462+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:15.463+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:15.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:15.623+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:15.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:15.623+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:15.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:15.623+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:15.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:15.624+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:16.371 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:11:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:11:16.205+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 24186 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:11:16.622 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:16.567+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:16.622 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:16.567+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:16.622 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:16.567+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:16.622 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:16.567+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:16.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:16.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:16.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:16.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:16.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:16.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:16.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:16.428+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:16.623 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:16.619+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:16.623 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:16.619+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:16.624 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:16.619+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:16.624 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:16.619+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:17.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:17.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:17.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:17.417+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:17.656+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:17.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:17.656+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:17.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:17.656+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:17.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:17.657+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:17.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:17.613+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:17.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:17.613+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:17.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:17.613+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:17.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:17.614+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:18.462+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:18.462+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:18.462+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:18.463+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:18.626+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:18.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:18.626+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:18.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:18.626+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:18.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:18.627+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:18.583+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:18.583+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:18.583+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:18.583+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:19.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:19.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:19.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:19.660+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:19.612+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:19.612+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:19.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:19.612+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:19.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:19.613+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:19.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:19.456+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:19.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:19.456+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:19.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:19.456+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:19.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:19.457+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:20.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:20.432+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:20.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:20.432+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:20.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:20.432+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:20.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:20.433+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:20.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:20.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:20.639+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:20.639+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:20.624+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:20.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:20.624+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:20.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:20.624+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:20.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:20.625+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:21.445 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:11:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:11:21.208+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 24277 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:11:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:21.654+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:21.654+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:21.654+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:21.654+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:21.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:21.673+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:21.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:21.673+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:21.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:21.673+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:21.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:21.674+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:21.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:21.443+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:21.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:21.443+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:21.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:21.443+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:21.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:21.443+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:22.627+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:22.627+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:22.627+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:22.628+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:22.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:22.683+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:22.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:22.683+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:22.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:22.683+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:22.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:22.684+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:22.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:22.486+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:22.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:22.486+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:22.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:22.486+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:22.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:22.487+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:23.516+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:23.516+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:23.516+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:23.517+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:23.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:23.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:23.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:23.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:23.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:23.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:23.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:23.580+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:23.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:23.660+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:23.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:23.660+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:23.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:23.660+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:23.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:23.660+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:24.550+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:24.550+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:24.550+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:24.551+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:24.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:24.614+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:24.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:24.614+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:24.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:24.614+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:24.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:24.615+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:24.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:24.641+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:24.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:24.641+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:24.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:24.641+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:24.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:24.641+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:25.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:25.606+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:25.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:25.606+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:25.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:25.606+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:25.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:25.606+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:25.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:25.636+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:25.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:25.636+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:25.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:25.636+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:25.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:25.637+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:25.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:25.537+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:25.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:25.537+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:25.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:25.537+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:25.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:25.537+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:26.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:11:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:11:26.211+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 24365 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:11:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:26.633+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:26.633+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:26.633+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:26.634+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:26.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:26.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:26.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:26.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:26.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:26.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:26.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:26.598+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:26.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:26.545+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:26.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:26.545+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:26.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:26.545+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:26.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:26.545+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:27.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:27.522+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:27.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:27.522+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:27.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:27.522+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:27.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:27.523+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:27.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:27.592+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:27.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:27.592+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:27.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:27.592+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:27.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:27.593+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:27.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:27.630+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:27.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:27.630+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:27.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:27.630+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:27.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:27.631+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:28.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:28.496+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:28.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:28.496+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:28.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:28.496+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:28.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:28.497+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:28.586+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:28.586+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:28.586+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:28.586+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:28.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:28.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:28.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:28.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:28.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:28.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:28.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:28.619+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:29.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:29.660+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:29.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:29.660+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:29.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:29.660+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:29.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:29.661+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:29.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:29.615+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:29.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:29.615+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:29.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:29.615+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:29.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:29.615+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:29.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:29.501+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:29.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:29.501+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:29.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:29.501+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:29.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:29.502+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:30.565+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:30.566+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:30.566+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:30.566+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:30.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:30.453+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:30.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:30.453+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:30.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:30.453+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:30.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:30.453+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:30.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:30.623+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:30.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:30.623+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:30.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:30.623+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:30.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:30.624+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:31.488 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:11:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:11:31.214+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 24457 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:11:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:31.486+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:31.486+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:31.486+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:31.486+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:31.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:31.562+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:31.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:31.562+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:31.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:31.562+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:31.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:31.563+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:31.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:31.607+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:31.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:31.607+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:31.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:31.607+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:31.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:31.607+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:32.444+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:32.444+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:32.444+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:32.445+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:32.577+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:32.577+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:32.577+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:32.577+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:32.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:32.621+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:32.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:32.621+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:32.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:32.621+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:32.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:32.621+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:33.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:33.670+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:33.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:33.670+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:33.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:33.670+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:33.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:33.671+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:33.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:33.443+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:33.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:33.443+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:33.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:33.443+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:33.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:33.444+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:33.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:33.601+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:33.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:33.601+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:33.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:33.601+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:33.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:33.602+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:34.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:34.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:34.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:34.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:34.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:34.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:34.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:34.419+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:34.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:34.596+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:34.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:34.596+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:34.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:34.596+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:34.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:34.597+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:34.685+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:34.685+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:34.685+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:34.685+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:35.657+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:35.657+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:35.657+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:35.658+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:35.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:35.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:35.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:35.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:35.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:35.417+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:35.561+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:35.561+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:35.561+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:35.561+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:36.474 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:11:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:11:36.217+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 24545 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:11:36.475 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:36.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:36.475 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:36.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:36.475 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:36.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:36.475 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:36.429+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:36.600+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:36.600+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:36.600+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:36.601+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:36.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:36.665+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:36.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:36.665+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:36.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:36.665+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:36.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:36.666+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:37.571+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:37.571+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:37.571+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:37.571+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:37.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:37.663+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:37.663+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:37.663+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:37.663+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:37.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:37.446+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:37.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:37.446+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:37.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:37.446+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:37.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:37.447+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:38.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:38.445+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:38.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:38.445+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:38.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:38.445+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:38.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:38.446+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:38.712 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:38.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:38.712 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:38.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:38.712 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:38.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:38.712 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:38.532+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:38.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:38.708+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:38.708+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:38.708+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:38.709+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:39.447+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:39.447+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:39.447+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:39.448+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:39.561+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:39.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:39.561+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:39.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:39.561+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:39.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:39.561+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:40.194 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:39.715+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:40.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:39.715+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:40.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:39.715+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:40.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:39.716+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:40.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:40.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:40.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:40.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:40.414+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:40.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:40.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:40.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:40.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:40.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:40.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:40.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:40.533+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:40.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:40.739+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:40.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:40.739+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:40.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:40.739+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:40.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:40.740+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:41.484 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:11:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:11:41.219+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 24636 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:11:41.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:41.405+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:41.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:41.405+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:41.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:41.405+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:41.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:41.406+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:41.774 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:41.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:41.774 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:41.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:41.774 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:41.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:41.774 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:41.523+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:42.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:41.771+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:41.771+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:41.771+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:41.772+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:42.502+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:42.502+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:42.502+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:42.503+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:42.371+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:42.371+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:42.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:42.371+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:42.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:42.371+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:42.723+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:42.723+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:42.723+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:42.723+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:43.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:43.355+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:43.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:43.355+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:43.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:43.355+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:43.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:43.356+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:43.618 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:43.549+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:43.619 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:43.549+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:43.619 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:43.549+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:43.619 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:43.550+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:43.725+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:43.725+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:43.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:43.725+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:43.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:43.726+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:44.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:44.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:44.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:44.504+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:44.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:44.404+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:44.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:44.404+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:44.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:44.404+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:44.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:44.405+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:44.738+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:44.738+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:44.738+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:44.738+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:45.501+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:45.501+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:45.501+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:45.502+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:45.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:45.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:45.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:45.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:45.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:45.388+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:46.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:45.736+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:45.736+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:45.736+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:45.737+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:46.517 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:11:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:11:46.222+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 24724 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:11:46.517 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:46.427+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:46.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:46.427+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:46.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:46.427+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:46.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:46.427+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:46.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:46.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:46.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:46.515+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:46.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:46.711+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:46.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:46.711+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:46.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:46.711+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:46.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:46.711+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:47.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:47.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:47.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:47.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:47.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:47.379+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:47.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:47.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:47.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:47.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:47.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:47.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:47.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:47.515+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:47.731+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:47.731+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:47.731+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:47.732+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:48.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:48.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:48.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:48.523+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:48.417+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:48.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:48.417+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:48.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:48.417+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:48.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:48.418+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:48.716+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:48.716+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:48.716+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:48.717+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:49.690 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:49.405+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:49.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:49.405+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:49.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:49.405+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:49.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:49.405+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:49.691 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:49.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:49.691 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:49.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:49.691 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:49.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:49.692 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:49.511+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:49.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:49.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:49.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:49.689+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:50.551+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:50.551+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:50.551+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:50.552+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:50.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:50.444+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:50.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:50.444+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:50.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:50.444+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:50.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:50.444+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:50.714+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:50.714+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:50.714+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:50.715+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:51.581 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:11:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:11:51.225+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 24816 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:11:51.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:51.438+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:51.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:51.438+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:51.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:51.438+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:51.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:51.438+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:51.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:51.759+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:51.759+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:51.759+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:51.760+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:51.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:51.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:51.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:51.579+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:51.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:51.580+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:52.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:52.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:52.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:52.600+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:52.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:52.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:52.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:52.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:52.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:52.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:52.436+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:52.781+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:52.781+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:52.781+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:52.781+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:53.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:53.596+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:53.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:53.596+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:53.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:53.596+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:53.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:53.597+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:53.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:53.424+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:53.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:53.424+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:53.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:53.424+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:53.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:53.425+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:53.787+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:53.787+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:53.787+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:53.787+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:54.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:54.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:54.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:54.574+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:54.434+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:54.434+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:54.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:54.434+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:54.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:54.435+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:54.822+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:54.822+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:54.822+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:54.823+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:55.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:55.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:55.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:55.532+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:55.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:55.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:55.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:55.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:55.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:55.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:55.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:55.409+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:56.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:55.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:55.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:55.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:55.834+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:56.513 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:11:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:11:56.226+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 24904 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:11:56.514 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:56.439+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:56.514 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:56.439+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:56.514 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:56.439+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:56.514 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:56.440+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:56.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:56.511+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:56.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:56.511+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:56.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:56.511+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:56.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:56.512+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:57.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:56.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:56.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:56.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:56.846+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:57.423+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:57.423+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:57.423+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:57.424+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:57.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:57.470+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:57.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:57.470+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:57.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:57.470+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:57.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:57.471+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:57.810+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:57.810+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:57.810+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:57.811+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:58.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:58.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:58.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:58.474+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:58.459+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:58.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:58.459+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:58.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:58.459+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:58.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:58.460+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:58.765+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:58.765+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:58.765+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:11:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:58.765+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:11:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:59.483+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:59.483+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:59.483+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:11:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:11:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:11:59.483+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:11:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:59.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:59.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:59.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:59.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:59.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:11:59.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:11:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:11:59.416+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:00.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:59.792+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:00.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:59.792+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:00.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:59.792+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:00.197 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:11:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:11:59.792+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:00.523+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:00.523+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:00.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:00.523+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:00.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:00.524+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:00.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:00.449+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:00.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:00.449+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:00.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:00.449+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:00.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:00.450+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:00.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:00.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:00.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:00.791+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:01.506 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:12:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:12:01.230+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 24995 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:12:01.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:01.433+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:01.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:01.433+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:01.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:01.433+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:01.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:01.434+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:01.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:01.503+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:01.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:01.503+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:01.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:01.503+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:01.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:01.503+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:02.064 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:01.799+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:02.064 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:01.799+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:02.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:01.799+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:02.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:01.799+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:02.405+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:02.405+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:02.405+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:02.406+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:02.551+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:02.551+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:02.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:02.551+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:02.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:02.552+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:03.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:02.787+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:02.787+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:02.787+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:02.788+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:03.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:03.527+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:03.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:03.527+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:03.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:03.527+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:03.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:03.527+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:03.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:03.436+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:03.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:03.436+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:03.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:03.436+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:03.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:03.437+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:03.738+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:03.738+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:03.738+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:03.739+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:04.436+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:04.436+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:04.436+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:04.437+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:04.542+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:04.542+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:04.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:04.542+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:04.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:04.543+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:05.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:04.775+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:05.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:04.775+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:05.137 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:04.775+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:05.137 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:04.776+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:05.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:05.398+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:05.398+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:05.398+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:05.399+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:05.555+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:05.555+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:05.555+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:05.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:05.556+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:05.788+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:05.788+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:05.788+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:05.789+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:06.607 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:12:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:12:06.233+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 25083 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:12:06.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:06.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:06.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:06.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:06.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:06.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:06.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:06.431+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:06.870 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:06.605+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:06.870 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:06.605+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:06.870 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:06.605+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:06.871 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:06.606+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:06.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:06.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:06.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:06.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:06.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:06.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:06.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:06.767+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:07.820 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:07.643+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:07.820 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:07.643+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:07.820 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:07.643+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:07.820 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:07.644+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:07.820 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:07.472+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:07.821 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:07.472+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:07.821 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:07.472+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:07.821 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:07.473+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:08.209 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:07.817+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:08.209 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:07.817+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:08.209 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:07.817+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:08.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:07.818+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:08.438+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:08.438+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:08.438+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:08.438+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:08.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:08.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:08.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:08.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:08.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:08.673+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:08.802+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:08.802+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:08.802+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:08.802+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:09.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:09.427+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:09.427+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:09.427+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:09.428+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:10.197 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:09.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:10.198 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:09.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:10.198 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:09.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:10.198 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:09.719+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:10.198 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:09.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:10.198 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:09.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:10.198 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:09.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:10.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:09.806+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:10.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:10.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:10.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:10.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:10.416+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:10.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:10.691+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:10.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:10.691+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:10.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:10.691+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:10.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:10.692+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:10.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:10.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:10.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:10.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:10.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:10.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:10.811+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:11.667 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:12:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:12:11.236+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 25175 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:12:11.667 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:11.458+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:11.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:11.458+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:11.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:11.458+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:11.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:11.458+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:11.947 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:11.665+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:11.947 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:11.665+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:11.947 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:11.665+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:11.947 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:11.666+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:11.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:11.769+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:11.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:11.769+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:11.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:11.769+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:11.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:11.770+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:12.652+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:12.652+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:12.652+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:12.653+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:12.440+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:12.440+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:12.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:12.440+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:12.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:12.440+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:12.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:12.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:12.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:12.752+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:13.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:13.632+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:13.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:13.632+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:13.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:13.632+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:13.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:13.633+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:13.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:13.463+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:13.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:13.463+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:13.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:13.463+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:13.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:13.464+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:13.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:13.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:13.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:13.728+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:14.589+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:14.589+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:14.589+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:14.590+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:14.465+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:14.465+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:14.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:14.465+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:14.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:14.465+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:14.769+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:14.769+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:14.769+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:14.770+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:15.819 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:15.559+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:15.819 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:15.559+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:15.819 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:15.559+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:15.819 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:15.560+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:15.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:15.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:15.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:15.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:15.820 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:15.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:15.820 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:15.479+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:15.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:15.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:15.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:15.817+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:16.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:12:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:12:16.239+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 25263 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:12:16.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:16.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:16.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:16.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:16.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:16.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:16.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:16.486+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:16.862 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:16.535+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:16.862 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:16.535+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:16.862 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:16.535+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:16.862 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:16.536+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:16.859+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:16.859+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:16.859+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:16.860+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:17.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:17.525+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:17.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:17.525+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:17.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:17.525+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:17.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:17.526+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:17.862 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:17.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:17.862 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:17.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:17.862 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:17.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:17.862 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:17.467+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:17.858+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:17.858+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:17.858+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:17.859+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:18.833 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:18.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:18.834 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:18.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:18.834 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:18.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:18.834 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:18.466+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:18.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:18.481+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:18.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:18.481+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:18.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:18.481+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:18.835 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:18.481+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:18.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:18.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:18.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:18.831+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:19.448+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:19.448+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:19.448+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:19.449+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:19.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:19.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:19.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:19.480+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:20.198 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:19.786+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:20.198 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:19.786+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:20.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:19.786+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:20.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:19.787+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:20.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:20.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:20.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:20.516+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:20.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:20.516+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:20.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:20.516+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:20.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:20.488+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:20.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:20.488+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:20.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:20.488+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:20.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:20.488+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:21.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:20.818+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:20.818+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:20.818+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:20.819+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:21.497 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:12:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:12:21.242+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 25354 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:12:21.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:21.495+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:21.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:21.495+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:21.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:21.495+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:21.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:21.495+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:21.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:21.506+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:21.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:21.506+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:21.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:21.506+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:21.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:21.507+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:22.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:21.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:22.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:21.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:22.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:21.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:22.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:21.867+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:22.449+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:22.449+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:22.449+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:22.450+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:22.555+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:22.555+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:22.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:22.555+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:22.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:22.556+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:23.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:22.860+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:22.860+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:22.860+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:22.861+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:23.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:23.454+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:23.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:23.454+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:23.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:23.454+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:23.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:23.455+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:23.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:23.539+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:23.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:23.539+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:23.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:23.539+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:23.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:23.539+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:24.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:23.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:23.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:23.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:23.816+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:24.456+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:24.456+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:24.456+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:24.457+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:24.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:24.584+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:24.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:24.584+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:24.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:24.584+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:24.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:24.584+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:25.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:24.815+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:24.815+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:24.815+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:24.815+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:25.461+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:25.461+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:25.461+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:25.462+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:25.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:25.578+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:25.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:25.578+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:25.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:25.578+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:25.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:25.579+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:26.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:25.862+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:25.862+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:25.862+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:25.863+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:26.588 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:12:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:12:26.245+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 25442 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:12:26.589 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:26.493+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:26.589 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:26.493+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:26.589 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:26.493+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:26.589 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:26.493+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:26.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:26.586+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:26.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:26.586+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:26.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:26.586+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:26.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:26.587+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:27.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:26.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:27.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:26.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:27.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:26.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:27.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:26.891+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:27.462+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:27.463+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:27.463+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:27.463+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:27.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:27.576+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:27.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:27.576+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:27.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:27.576+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:27.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:27.577+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:27.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:27.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:27.845+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:27.846+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:28.462+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:28.462+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:28.462+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:28.463+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:28.554+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:28.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:28.554+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:28.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:28.554+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:28.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:28.555+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:28.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:28.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:28.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:28.870+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:29.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:29.455+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:29.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:29.455+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:29.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:29.455+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:29.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:29.455+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:29.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:29.596+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:29.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:29.596+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:29.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:29.596+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:29.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:29.597+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:30.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:29.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:30.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:29.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:30.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:29.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:30.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:29.892+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:30.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:30.572+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:30.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:30.572+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:30.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:30.572+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:30.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:30.573+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:30.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:30.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:30.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:30.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:30.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:30.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:30.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:30.483+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:30.931+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:30.931+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:30.931+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:30.932+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:31.571 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:12:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:12:31.248+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 25534 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:12:31.571 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:31.491+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:31.571 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:31.491+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:31.571 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:31.491+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:31.571 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:31.491+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:31.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:31.568+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:31.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:31.568+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:31.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:31.568+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:31.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:31.569+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:31.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:31.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:31.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:31.942+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:32.552 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:32.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:32.552 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:32.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:32.553 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:32.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:32.553 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:32.480+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:32.913 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:32.550+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:32.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:32.550+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:32.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:32.550+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:32.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:32.550+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:32.911+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:32.911+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:32.911+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:32.912+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:33.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:33.597+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:33.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:33.597+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:33.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:33.597+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:33.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:33.598+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:33.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:33.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:33.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:33.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:33.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:33.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:33.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:33.516+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:33.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:33.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:33.870+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:33.871+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:34.875 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:34.552+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:34.875 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:34.552+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:34.875 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:34.552+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:34.876 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:34.553+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:34.876 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:34.533+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:34.876 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:34.533+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:34.876 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:34.533+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:34.876 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:34.534+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:34.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:34.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:34.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:34.873+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:35.856 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:35.572+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:35.857 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:35.572+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:35.857 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:35.572+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:35.857 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:35.572+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:35.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:35.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:35.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:35.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:35.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:35.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:35.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:35.573+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:35.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:35.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:35.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:35.872+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:36.549 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:12:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:12:36.251+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 25622 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:12:36.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:36.546+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:36.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:36.546+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:36.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:36.547+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:36.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:36.547+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:36.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:36.619+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:36.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:36.619+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:36.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:36.619+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:36.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:36.620+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:36.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:36.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:36.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:36.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:36.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:36.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:36.872+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:37.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:37.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:37.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:37.830+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:37.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:37.614+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:37.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:37.614+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:37.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:37.614+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:37.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:37.615+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:37.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:37.582+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:37.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:37.582+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:37.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:37.582+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:37.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:37.583+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:38.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:38.590+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:38.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:38.590+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:38.880 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:38.590+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:38.880 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:38.590+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:38.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:38.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:38.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:38.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:38.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:38.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:38.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:38.587+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:38.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:38.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:38.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:38.877+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:39.870 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:39.605+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:39.871 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:39.605+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:39.871 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:39.605+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:39.871 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:39.605+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:39.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:39.553+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:39.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:39.553+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:39.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:39.553+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:39.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:39.554+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:40.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:39.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:40.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:39.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:40.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:39.868+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:40.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:39.868+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:40.888 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:40.598+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:40.888 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:40.598+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:40.888 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:40.598+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:40.888 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:40.599+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:40.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:40.540+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:40.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:40.540+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:40.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:40.540+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:40.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:40.540+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:41.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:40.885+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:40.885+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:40.885+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:40.886+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:41.572 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:12:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:12:41.254+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 25713 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:12:41.865 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:41.601+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:41.865 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:41.601+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:41.865 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:41.601+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:41.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:41.602+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:41.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:41.569+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:41.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:41.569+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:41.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:41.569+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:41.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:41.570+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:42.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:41.894+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:41.894+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:41.894+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:41.895+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:42.623+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:42.623+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:42.623+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:42.624+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:42.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:42.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:42.599+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:42.600+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:42.875+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:42.875+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:42.875+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:42.875+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:43.854 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:43.592+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:43.854 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:43.592+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:43.854 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:43.592+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:43.854 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:43.593+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:43.855 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:43.569+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:43.855 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:43.569+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:43.855 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:43.569+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:43.855 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:43.570+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:44.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:43.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:43.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:43.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:43.851+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:44.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:44.575+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:44.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:44.575+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:44.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:44.575+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:44.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:44.575+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:44.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:44.592+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:44.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:44.592+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:44.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:44.592+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:44.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:44.592+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:44.901+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:44.901+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:44.901+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:44.901+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:45.878 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:45.572+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:45.878 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:45.572+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:45.879 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:45.572+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:45.879 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:45.573+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:45.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:45.535+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:45.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:45.535+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:45.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:45.535+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:45.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:45.536+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:45.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:45.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:45.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:45.873+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:46.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:12:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:12:46.256+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 25801 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:12:46.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:46.517+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:46.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:46.517+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:46.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:46.517+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:46.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:46.517+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:46.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:46.585+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:46.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:46.585+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:46.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:46.585+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:46.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:46.586+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:46.920+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:46.920+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:46.920+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:46.921+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:47.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:47.555+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:47.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:47.555+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:47.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:47.555+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:47.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:47.555+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:47.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:47.529+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:47.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:47.529+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:47.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:47.529+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:47.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:47.529+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:48.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:47.947+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:47.947+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:47.947+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:47.948+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:48.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:48.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:48.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:48.533+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:48.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:48.489+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:48.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:48.489+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:48.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:48.489+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:48.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:48.490+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:49.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:48.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:48.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:48.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:48.980+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:49.501+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:49.501+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:49.501+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:49.502+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:49.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:49.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:49.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:49.506+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:49.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:49.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:49.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:49.967+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:50.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:50.527+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:50.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:50.527+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:50.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:50.527+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:50.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:50.527+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:50.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:50.529+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:50.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:50.529+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:50.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:50.529+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:50.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:50.530+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:51.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:50.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:50.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:50.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:50.929+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:51.545 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:12:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:12:51.260+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 25893 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:12:51.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:51.551+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:51.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:51.551+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:51.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:51.551+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:51.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:51.552+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:51.862 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:51.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:51.862 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:51.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:51.862 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:51.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:51.862 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:51.544+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:52.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:51.888+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:51.888+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:51.888+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:51.889+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:52.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:52.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:52.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:52.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:52.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:52.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:52.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:52.507+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:52.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:52.530+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:52.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:52.530+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:52.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:52.530+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:52.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:52.531+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:52.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:52.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:52.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:52.917+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:53.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:53.503+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:53.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:53.503+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:53.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:53.503+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:53.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:53.504+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:53.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:53.542+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:53.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:53.542+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:53.893 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:53.542+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:53.893 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:53.543+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:53.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:53.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:53.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:53.890+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:54.517+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:54.517+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:54.517+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:54.518+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:54.522+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:54.522+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:54.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:54.522+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:54.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:54.522+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:54.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:54.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:54.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:54.874+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:55.829 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:55.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:55.829 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:55.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:55.829 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:55.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:55.829 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:55.511+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:55.829 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:55.562+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:55.829 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:55.562+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:55.830 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:55.562+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:55.830 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:55.563+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:56.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:55.826+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:55.826+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:55.826+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:55.827+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:56.525 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:12:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:12:56.263+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 25981 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:12:56.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:56.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:56.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:56.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:56.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:56.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:56.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:56.523+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:56.810 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:56.552+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:56.810 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:56.552+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:56.810 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:56.552+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:56.810 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:56.552+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:57.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:56.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:56.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:56.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:56.807+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:57.808 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:57.522+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:57.809 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:57.522+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:57.809 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:57.522+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:57.809 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:57.523+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:57.810 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:57.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:57.810 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:57.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:57.810 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:57.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:57.810 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:57.483+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:57.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:57.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:57.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:57.806+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:58.557+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:58.557+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:58.557+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:58.558+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:58.452+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:58.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:58.452+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:58.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:58.452+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:58.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:58.452+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:58.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:58.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:58.772+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:12:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:58.772+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:12:59.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:59.548+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:59.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:59.548+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:59.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:59.548+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:12:59.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:12:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:12:59.549+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:12:59.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:59.494+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:59.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:59.494+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:59.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:59.494+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:12:59.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:12:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:12:59.494+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:00.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:59.756+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:00.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:59.756+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:00.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:59.756+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:00.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:12:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:12:59.756+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:00.752 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:00.516+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:00.752 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:00.516+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:00.752 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:00.516+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:00.752 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:00.516+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:00.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:00.477+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:00.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:00.477+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:00.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:00.477+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:00.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:00.477+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:00.749+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:00.749+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:00.749+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:00.750+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:01.526 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:13:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:13:01.266+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 26072 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:13:01.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:01.506+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:01.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:01.506+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:01.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:01.506+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:01.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:01.507+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:01.796 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:01.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:01.796 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:01.524+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:01.796 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:01.524+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:01.796 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:01.524+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:02.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:01.793+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:01.793+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:01.793+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:01.794+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:02.825 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:02.515+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:02.825 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:02.515+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:02.825 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:02.515+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:02.825 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:02.516+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:02.826 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:02.483+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:02.826 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:02.483+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:02.826 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:02.483+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:02.826 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:02.483+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:02.823+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:02.823+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:02.823+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:02.823+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:03.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:03.433+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:03.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:03.433+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:03.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:03.433+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:03.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:03.434+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:03.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:03.535+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:03.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:03.535+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:03.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:03.535+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:03.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:03.536+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:03.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:03.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:03.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:03.874+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:04.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:04.559+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:04.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:04.559+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:04.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:04.559+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:04.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:04.559+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:04.840 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:04.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:04.840 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:04.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:04.840 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:04.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:04.840 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:04.467+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:04.836+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:04.836+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:04.836+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:04.837+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:05.803 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:05.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:05.803 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:05.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:05.803 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:05.473+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:05.803 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:05.474+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:05.804 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:05.592+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:05.804 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:05.592+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:05.804 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:05.592+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:05.804 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:05.593+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:05.800+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:05.800+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:05.800+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:05.801+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:06.576 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:13:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:13:06.269+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 26160 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:13:06.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:06.488+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:06.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:06.488+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:06.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:06.488+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:06.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:06.489+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:06.798+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:06.798+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:06.798+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:06.798+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:06.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:06.638+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:06.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:06.638+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:06.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:06.638+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:06.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:06.638+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:07.800 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:07.605+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:07.800 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:07.605+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:07.801 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:07.605+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:07.801 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:07.605+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:07.801 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:07.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:07.801 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:07.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:07.801 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:07.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:07.801 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:07.531+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:07.802 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:07.776+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:07.802 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:07.776+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:07.802 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:07.776+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:07.802 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:07.776+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:08.744+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:08.744+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:08.744+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:08.745+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:08.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:08.650+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:08.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:08.650+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:08.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:08.650+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:08.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:08.651+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:08.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:08.521+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:08.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:08.521+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:08.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:08.521+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:08.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:08.521+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:09.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:09.675+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:09.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:09.675+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:09.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:09.675+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:09.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:09.675+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:09.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:09.495+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:09.784 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:09.495+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:09.784 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:09.495+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:09.784 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:09.496+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:10.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:09.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:10.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:09.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:10.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:09.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:10.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:09.781+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:10.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:10.684+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:10.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:10.684+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:10.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:10.684+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:10.799 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:10.685+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:10.799 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:10.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:10.799 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:10.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:10.799 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:10.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:10.799 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:10.526+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:11.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:10.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:10.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:10.795+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:10.796+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:11.679 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:13:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:13:11.272+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 26252 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:13:11.679 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:11.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:11.679 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:11.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:11.679 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:11.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:11.679 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:11.504+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:11.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:11.676+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:11.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:11.676+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:11.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:11.676+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:11.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:11.677+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:11.788+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:11.788+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:11.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:11.788+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:11.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:11.789+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:12.742 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:12.631+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:12.742 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:12.631+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:12.743 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:12.631+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:12.743 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:12.632+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:12.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:12.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:12.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:12.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:12.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:12.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:12.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:12.476+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:12.739+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:12.740+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:12.740+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:12.741+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:13.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:13.607+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:13.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:13.607+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:13.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:13.607+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:13.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:13.608+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:13.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:13.434+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:13.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:13.434+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:13.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:13.434+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:13.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:13.435+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:13.785+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:13.785+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:13.785+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:13.786+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:14.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:14.649+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:14.649+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:14.649+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:14.650+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:14.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:14.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:14.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:14.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:14.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:14.441+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:14.798+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:14.798+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:14.798+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:14.798+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:15.671+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:15.671+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:15.671+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:15.672+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:15.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:15.444+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:15.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:15.444+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:15.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:15.444+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:15.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:15.445+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:15.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:15.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:15.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:15.845+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:16.633 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:13:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:13:16.275+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 26340 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:13:16.634 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:16.426+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:16.634 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:16.426+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:16.634 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:16.426+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:16.634 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:16.427+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:16.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:16.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:16.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:16.831+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:16.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:16.631+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:16.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:16.631+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:16.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:16.631+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:16.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:16.632+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:17.396+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:17.396+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:17.396+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:17.397+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:17.641+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:17.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:17.641+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:17.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:17.641+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:17.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:17.642+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:17.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:17.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:17.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:17.842+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:18.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:18.364+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:18.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:18.364+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:18.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:18.364+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:18.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:18.365+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:18.657+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:18.657+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:18.657+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:18.658+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:18.834+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:18.834+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:18.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:18.834+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:18.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:18.835+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:19.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:19.329+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:19.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:19.329+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:19.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:19.329+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:19.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:19.330+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:19.650+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:19.650+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:19.650+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:19.651+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:19.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:19.790+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:19.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:19.790+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:19.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:19.790+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:19.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:19.791+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:20.341+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:20.341+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:20.341+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:20.342+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:20.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:20.670+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:20.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:20.670+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:20.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:20.670+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:20.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:20.670+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:20.778+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:20.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:20.778+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:20.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:20.778+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:20.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:20.778+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:21.702 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:13:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:13:21.279+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 26431 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:13:21.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:21.327+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:21.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:21.327+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:21.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:21.327+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:21.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:21.328+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:21.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:21.699+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:21.699+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:21.699+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:21.700+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:21.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:21.743+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:21.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:21.743+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:21.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:21.743+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:21.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:21.744+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:22.296+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:22.296+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:22.296+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:22.296+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:23.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:22.725+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:23.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:22.725+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:23.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:22.725+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:23.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:22.726+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:23.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:22.771+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:23.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:22.771+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:23.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:22.771+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:23.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:22.772+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:23.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:23.262+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:23.262+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:23.262+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:23.263+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:23.710+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:23.710+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:23.710+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:23.711+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:23.774+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:23.774+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:23.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:23.774+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:23.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:23.774+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:24.687 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:24.239+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:24.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:24.239+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:24.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:24.239+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:24.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:24.239+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:24.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:24.685+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:24.685+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:24.685+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:24.685+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:24.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:24.822+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:24.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:24.822+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:24.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:24.822+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:24.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:24.822+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:25.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:25.269+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:25.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:25.269+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:25.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:25.269+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:25.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:25.269+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:25.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:25.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:25.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:25.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:25.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:25.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:25.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:25.838+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:25.692+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:25.692+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:25.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:25.692+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:25.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:25.693+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:26.701 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:13:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:13:26.282+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 26519 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:13:26.701 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:26.304+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:26.701 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:26.304+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:26.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:26.304+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:26.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:26.304+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:26.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:26.698+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:26.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:26.698+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:26.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:26.698+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:26.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:26.699+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:26.848+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:26.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:26.848+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:26.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:26.848+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:26.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:26.849+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:27.260+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:27.260+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:27.260+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:27.260+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:28.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:27.737+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:28.082 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:27.737+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:28.082 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:27.737+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:28.082 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:27.738+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:28.082 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:27.829+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:28.083 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:27.829+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:28.083 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:27.829+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:28.083 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:27.829+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:28.280+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:28.280+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:28.280+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:28.280+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:29.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:28.745+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:29.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:28.745+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:29.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:28.745+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:29.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:28.745+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:29.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:28.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:29.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:28.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:29.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:28.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:29.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:28.784+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:29.254+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:29.254+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:29.254+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:29.254+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:30.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:29.714+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:30.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:29.714+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:30.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:29.714+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:30.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:29.714+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:30.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:29.817+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:30.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:29.817+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:30.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:29.817+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:30.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:29.817+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:30.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:30.277+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:30.277+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:30.277+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:30.278+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:31.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:30.754+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:31.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:30.754+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:31.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:30.754+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:31.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:30.755+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:31.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:30.853+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:31.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:30.853+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:31.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:30.853+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:31.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:30.854+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:31.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:13:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:13:31.284+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 26611 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:13:31.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:31.315+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:31.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:31.315+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:31.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:31.315+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:31.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:31.315+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:31.763+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:31.763+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:31.763+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:31.764+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:31.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:32.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:31.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:32.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:31.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:32.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:31.867+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:32.292+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:32.292+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:32.292+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:32.293+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:33.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:32.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:33.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:32.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:33.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:32.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:33.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:32.780+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:33.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:32.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:33.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:32.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:33.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:32.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:33.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:32.889+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:33.249+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:33.249+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:33.249+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:33.250+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:33.819+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:33.819+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:33.819+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:33.819+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:33.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:33.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:33.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:33.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:33.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:33.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:33.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:33.872+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:34.288+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:34.288+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:34.288+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:34.288+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:34.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:34.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:34.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:34.791+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:35.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:34.897+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:35.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:34.897+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:35.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:34.897+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:35.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:34.898+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:35.258+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:35.258+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:35.258+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:35.258+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:35.931+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:35.931+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:35.931+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:35.932+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:35.757+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:35.757+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:36.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:35.757+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:36.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:35.758+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:36.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:13:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:13:36.289+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 26699 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:13:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:36.302+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:36.302+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:36.302+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:36.302+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:36.951+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:36.951+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:36.951+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:36.952+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:37.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:36.795+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:37.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:36.795+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:37.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:36.795+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:37.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:36.795+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:37.265+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:37.265+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:37.265+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:37.265+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:37.948+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:37.948+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:37.948+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:37.949+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:37.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:38.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:37.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:38.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:37.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:38.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:37.819+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:38.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:38.222+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:38.222+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:38.222+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:38.222+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:39.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:38.788+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:39.234 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:38.788+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:39.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:38.788+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:39.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:38.788+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:39.235 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:38.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:39.235 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:38.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:39.235 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:38.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:39.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:38.989+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:39.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:39.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:39.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:39.188+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:40.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:40.022+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:40.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:40.022+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:40.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:40.022+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:40.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:40.022+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:40.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:39.823+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:40.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:39.823+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:40.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:39.823+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:40.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:39.823+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:40.144+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:40.144+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:40.144+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:40.145+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:41.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:40.837+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:41.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:40.837+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:41.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:40.837+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:41.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:40.837+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:41.294 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:41.134+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:41.295 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:41.134+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:41.295 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:41.134+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:41.295 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:41.135+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:41.295 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:41.008+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:41.295 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:41.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:41.296 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:41.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:41.296 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:41.009+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:41.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:13:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:13:41.292+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 26789 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:13:42.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:41.800+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:42.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:41.800+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:42.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:41.800+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:42.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:41.801+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:42.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:42.012+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:42.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:42.012+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:42.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:42.012+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:42.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:42.012+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:42.145+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:42.145+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:42.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:42.145+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:42.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:42.145+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:43.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:42.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:43.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:42.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:43.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:42.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:43.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:42.840+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:43.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:42.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:43.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:42.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:43.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:42.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:43.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:42.968+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:43.441 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:43.157+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:43.441 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:43.157+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:43.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:43.157+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:43.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:43.158+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:43.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:43.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:43.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:43.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:43.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:43.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:43.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:43.845+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:44.003+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:44.003+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:44.003+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:44.004+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:44.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:44.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:44.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:44.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:44.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:44.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:44.148+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:45.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:44.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:45.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:44.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:45.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:44.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:45.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:44.800+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:45.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:45.030+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:45.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:45.030+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:45.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:45.030+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:45.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:45.030+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:45.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:45.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:45.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:45.172+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:46.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:45.847+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:46.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:45.847+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:46.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:45.847+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:46.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:45.848+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:46.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:46.011+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:46.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:46.011+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:46.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:46.011+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:46.163 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:46.012+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:46.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:13:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:13:46.295+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 26876 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:13:46.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:46.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:46.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:46.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:46.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:46.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:46.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:46.159+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:47.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:46.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:47.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:46.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:47.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:46.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:47.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:46.879+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:47.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:47.052+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:47.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:47.052+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:47.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:47.052+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:47.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:47.053+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:47.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:47.167+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:47.167+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:47.167+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:47.168+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:48.146+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:48.146+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:48.146+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:48.146+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:47.919+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:48.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:47.919+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:48.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:47.919+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:48.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:47.919+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:48.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:48.017+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:48.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:48.017+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:48.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:48.017+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:48.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:48.017+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:49.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:49.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:49.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:49.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:49.159+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:48.927+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:48.927+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:49.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:48.927+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:49.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:48.928+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:49.013+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:49.013+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:49.013+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:49.013+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:50.204+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:50.204+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:50.204+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:50.204+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:50.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:49.970+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:50.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:49.970+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:50.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:49.970+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:50.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:49.970+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:50.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:49.976+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:50.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:49.976+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:50.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:49.976+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:50.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:49.976+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:51.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:50.968+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:51.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:50.968+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:51.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:50.968+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:51.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:50.969+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:51.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:50.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:51.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:50.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:51.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:50.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:51.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:50.934+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:51.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:13:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:13:51.298+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 26967 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:13:51.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:51.170+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:51.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:51.170+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:51.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:51.170+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:51.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:51.170+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:52.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:51.979+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:52.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:51.979+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:52.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:51.979+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:52.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:51.979+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:52.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:51.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:52.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:51.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:52.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:51.895+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:52.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:51.895+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:52.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:52.151+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:52.151+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:52.151+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:52.151+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:52.999+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:52.999+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:52.999+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:53.000+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:53.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:53.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:53.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:53.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:53.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:53.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:53.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:53.163+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:53.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:52.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:53.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:52.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:53.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:52.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:53.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:52.934+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:53.923+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:53.923+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:53.923+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:53.924+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:54.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:53.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:54.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:53.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:54.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:53.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:54.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:53.989+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:54.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:54.123+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:54.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:54.123+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:54.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:54.123+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:54.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:54.124+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:54.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:54.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:54.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:54.959+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:55.114+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:55.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:55.114+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:55.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:55.114+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:55.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:55.114+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:54.920+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:54.920+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:54.920+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:54.921+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:56.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:55.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:56.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:55.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:56.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:55.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:56.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:55.948+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:56.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:55.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:56.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:55.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:56.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:55.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:56.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:55.880+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:56.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:13:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:13:56.301+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 27055 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:13:56.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:56.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:56.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:56.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:56.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:56.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:56.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:56.163+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:57.200 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:56.917+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:57.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:56.917+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:57.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:56.917+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:57.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:56.918+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:57.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:56.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:57.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:56.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:57.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:56.879+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:57.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:56.880+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:57.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:57.198+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:57.198+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:57.198+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:57.199+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:58.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:57.908+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:58.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:57.908+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:58.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:57.908+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:58.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:57.909+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:58.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:57.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:58.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:57.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:58.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:57.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:58.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:57.927+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:58.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:58.198+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:58.198+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:58.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:58.199+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:59.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:58.876+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:59.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:58.876+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:59.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:58.876+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:13:59.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:58.877+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:13:59.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:58.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:59.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:58.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:59.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:58.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:13:59.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:58.932+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:13:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:59.247+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:59.247+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:59.247+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:13:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:13:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:13:59.247+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:00.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:59.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:00.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:59.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:00.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:59.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:00.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:13:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:13:59.848+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:00.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:59.896+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:00.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:59.896+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:00.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:59.896+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:00.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:13:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:13:59.896+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:00.205+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:00.205+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:00.205+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:00.206+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:00.861+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:00.861+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:00.861+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:00.862+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:00.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:01.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:00.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:01.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:00.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:01.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:00.930+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:01.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:14:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:14:01.304+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 27146 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:14:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:01.254+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:01.254+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:01.254+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:01.254+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:01.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:01.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:01.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:01.888+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:02.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:01.892+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:02.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:01.892+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:02.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:01.892+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:02.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:01.893+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:02.298+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:02.298+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:02.298+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:02.299+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:02.897+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:02.897+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:02.897+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:02.897+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:02.877+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:03.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:02.877+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:03.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:02.877+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:03.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:02.878+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:03.626 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:03.314+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:03.626 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:03.314+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:03.626 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:03.314+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:03.626 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:03.315+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:03.890 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:03.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:03.890 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:03.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:03.890 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:03.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:03.890 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:03.851+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:04.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:03.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:03.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:03.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:03.888+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:04.284+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:04.284+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:04.284+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:04.285+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:04.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:04.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:04.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:04.881+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:04.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:04.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:04.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:04.829+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:05.263+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:05.263+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:05.263+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:05.264+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:05.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:05.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:05.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:05.840+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:05.825+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:05.825+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:05.825+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:05.826+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:06.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:14:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:14:06.308+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 27234 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:14:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:06.290+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:06.290+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:06.290+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:06.290+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:07.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:06.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:06.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:06.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:06.855+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:06.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:06.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:07.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:06.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:07.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:06.873+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:07.286+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:07.286+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:07.286+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:07.287+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:07.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:07.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:07.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:07.868+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:07.832+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:08.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:07.832+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:08.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:07.832+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:08.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:07.833+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:08.250+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:08.250+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:08.250+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:08.250+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:08.827+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:08.827+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:08.827+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:09.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:08.828+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:09.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:08.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:09.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:08.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:09.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:08.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:09.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:08.835+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:09.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:09.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:09.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:09.229+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:10.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:09.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:10.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:09.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:10.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:09.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:10.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:09.797+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:10.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:09.852+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:10.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:09.852+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:10.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:09.852+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:10.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:09.852+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:10.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:10.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:10.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:10.229+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:11.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:10.832+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:11.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:10.832+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:11.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:10.832+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:11.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:10.832+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:11.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:10.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:11.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:10.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:11.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:10.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:11.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:10.791+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:11.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:14:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:14:11.311+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 27326 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:14:11.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:11.238+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:11.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:11.238+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:11.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:11.238+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:11.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:11.239+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:12.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:11.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:12.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:11.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:12.203 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:11.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:12.203 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:11.752+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:12.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:11.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:12.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:11.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:12.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:11.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:12.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:11.838+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:12.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:12.200+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:12.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:12.200+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:12.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:12.200+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:12.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:12.201+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:13.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:12.745+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:13.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:12.745+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:13.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:12.745+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:13.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:12.746+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:13.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:12.803+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:13.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:12.803+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:13.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:12.803+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:13.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:12.804+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:13.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:13.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:13.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:13.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:13.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:13.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:13.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:13.189+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:13.728+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:13.728+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:13.728+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:13.729+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:13.843+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:13.843+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:13.843+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:13.844+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:14.190+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:14.190+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:14.190+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:14.191+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:15.195 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:14.734+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:15.195 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:14.734+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:15.195 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:14.734+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:15.196 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:14.735+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:15.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:14.804+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:15.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:14.804+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:15.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:14.804+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:15.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:14.805+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:15.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:15.192+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:15.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:15.192+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:15.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:15.192+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:15.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:15.193+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:16.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:15.778+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:16.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:15.778+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:16.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:15.778+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:16.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:15.779+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:16.149 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:15.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:16.149 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:15.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:16.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:15.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:16.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:15.822+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:16.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:14:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:14:16.314+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 27413 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:14:16.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:16.145+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:16.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:16.145+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:16.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:16.145+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:16.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:16.146+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:17.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:16.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:17.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:16.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:17.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:16.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:17.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:16.823+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:17.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:16.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:17.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:16.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:17.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:16.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:17.123 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:16.780+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:17.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:17.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:17.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:17.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:17.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:17.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:17.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:17.118+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:18.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:17.868+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:18.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:17.868+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:18.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:17.868+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:18.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:17.869+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:18.085 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:17.764+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:18.085 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:17.764+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:18.085 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:17.764+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:18.085 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:17.764+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:18.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:18.080+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:18.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:18.080+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:18.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:18.080+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:18.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:18.081+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:19.096 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:18.782+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:19.096 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:18.782+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:19.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:18.782+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:19.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:18.783+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:19.097 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:18.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:19.097 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:18.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:19.097 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:18.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:19.098 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:18.840+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:19.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:19.092+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:19.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:19.092+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:19.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:19.092+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:19.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:19.093+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:20.088 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:19.833+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:20.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:19.833+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:20.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:19.833+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:20.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:19.834+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:20.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:19.793+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:20.090 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:19.793+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:20.090 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:19.793+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:20.090 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:19.794+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:20.086+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:20.086+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:20.086+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:20.086+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:21.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:20.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:21.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:20.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:21.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:20.807+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:21.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:20.807+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:21.075 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:20.758+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:21.075 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:20.758+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:21.075 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:20.758+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:21.075 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:20.759+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:21.331 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:14:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:14:21.318+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 27504 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:14:21.331 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:21.072+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:21.331 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:21.072+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:21.331 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:21.072+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:21.332 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:21.072+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:22.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:21.775+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:22.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:21.775+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:22.093 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:21.775+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:22.093 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:21.775+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:22.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:21.785+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:22.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:21.785+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:22.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:21.785+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:22.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:21.786+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:22.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:22.090+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:22.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:22.090+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:22.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:22.090+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:22.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:22.091+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:23.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:22.742+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:23.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:22.742+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:23.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:22.742+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:23.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:22.743+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:23.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:22.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:23.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:22.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:23.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:22.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:23.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:22.812+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:23.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:23.126+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:23.126+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:23.126+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:23.127+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:23.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:23.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:23.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:23.828+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:23.712+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:23.712+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:23.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:23.712+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:23.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:23.713+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:24.092+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:24.092+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:24.092+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:24.093+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:25.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:24.743+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:25.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:24.743+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:25.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:24.743+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:25.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:24.743+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:25.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:24.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:25.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:24.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:25.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:24.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:25.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:24.836+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:25.132+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:25.132+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:25.132+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:25.132+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:26.100 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:25.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:26.100 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:25.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:26.100 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:25.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:26.100 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:25.741+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:26.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:25.797+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:26.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:25.797+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:26.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:25.797+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:26.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:25.798+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:26.365 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:14:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:14:26.321+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 27592 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:14:26.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:26.097+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:26.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:26.097+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:26.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:26.097+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:26.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:26.098+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:27.104 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:26.783+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:27.104 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:26.783+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:27.104 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:26.783+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:27.104 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:26.784+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:27.104 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:26.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:27.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:26.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:27.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:26.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:27.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:26.807+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:27.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:27.101+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:27.101+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:27.101+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:27.102+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:28.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:27.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:28.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:27.749+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:28.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:27.749+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:28.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:27.750+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:28.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:27.790+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:28.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:27.790+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:28.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:27.790+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:28.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:27.790+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:28.138+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:28.138+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:28.138+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:28.139+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:28.710+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:28.710+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:28.710+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:28.711+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:28.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:28.797+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:28.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:28.797+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:28.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:28.797+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:28.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:28.798+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:29.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:29.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:29.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:29.118+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:29.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:29.662+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:29.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:29.662+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:29.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:29.662+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:29.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:29.662+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:29.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:29.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:29.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:29.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:29.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:29.751+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:29.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:29.751+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:30.103+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:30.103+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:30.103+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:30.104+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:30.703+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:30.703+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:30.703+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:30.704+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:30.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:30.744+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:30.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:30.744+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:30.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:30.744+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:30.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:30.744+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:31.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:31.068+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:31.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:31.068+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:31.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:31.068+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:31.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:31.069+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:31.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:14:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:14:31.324+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 27684 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:14:32.057 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:31.747+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:32.057 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:31.747+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:32.058 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:31.747+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:32.058 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:31.748+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:32.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:31.725+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:32.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:31.725+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:32.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:31.725+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:32.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:31.726+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:32.054+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:32.054+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:32.054+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:32.055+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:32.717+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:32.717+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:32.717+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:32.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:32.718+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:32.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:32.704+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:32.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:32.704+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:32.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:32.704+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:32.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:32.704+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:33.087+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:33.087+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:33.087+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:33.087+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:33.726+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:33.726+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:33.726+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:33.727+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:33.655+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:33.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:33.655+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:33.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:33.655+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:33.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:33.656+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:34.084+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:34.084+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:34.084+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:34.085+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:34.685+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:34.685+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:34.685+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:34.686+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:34.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:34.686+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:34.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:34.686+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:34.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:34.686+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:34.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:34.686+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:35.040+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:35.040+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:35.040+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:35.040+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:35.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:35.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:35.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:35.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:35.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:35.688+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:35.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:35.689+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:35.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:35.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:35.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:35.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:35.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:35.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:35.728+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:36.330 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:36.067+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:36.330 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:36.067+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:36.330 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:36.067+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:36.331 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:36.067+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:36.682 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:14:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:14:36.327+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 27772 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:14:36.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:36.679+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:36.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:36.679+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:36.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:36.679+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:36.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:36.680+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:36.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:36.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:36.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:36.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:36.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:36.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:36.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:36.727+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:37.031+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:37.031+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:37.031+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:37.032+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:37.687+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:37.687+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:37.687+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:37.688+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:37.680+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:37.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:37.680+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:37.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:37.680+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:37.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:37.681+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:38.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:37.991+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:38.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:37.991+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:38.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:37.991+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:38.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:37.992+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:38.690+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:38.690+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:38.690+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:38.691+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:38.704+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:38.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:38.704+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:38.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:38.704+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:38.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:38.705+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:39.003+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:39.003+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:39.003+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:39.003+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:39.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:39.653+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:39.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:39.653+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:39.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:39.653+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:39.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:39.654+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:39.700+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:39.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:39.700+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:39.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:39.700+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:39.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:39.701+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:40.021+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:40.021+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:40.021+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:40.022+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:40.643+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:40.643+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:40.643+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:40.643+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:40.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:40.661+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:40.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:40.661+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:40.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:40.661+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:40.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:40.661+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:41.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:40.983+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:41.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:40.983+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:41.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:40.983+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:41.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:40.983+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:41.623 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:14:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:14:41.331+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 27863 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:14:41.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:41.684+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:41.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:41.684+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:41.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:41.684+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:41.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:41.685+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:41.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:41.620+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:41.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:41.620+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:41.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:41.620+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:41.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:41.621+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:42.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:41.991+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:41.991+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:41.991+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:41.992+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:42.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:42.723+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:42.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:42.723+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:42.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:42.723+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:42.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:42.724+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:42.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:42.664+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:42.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:42.664+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:42.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:42.664+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:42.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:42.665+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:42.973+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:42.973+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:42.973+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:42.974+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:43.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:43.674+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:43.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:43.674+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:43.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:43.674+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:43.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:43.675+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:43.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:43.636+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:43.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:43.636+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:43.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:43.636+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:43.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:43.636+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:44.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:43.960+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:43.960+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:43.960+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:43.961+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:44.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:44.637+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:44.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:44.637+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:44.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:44.637+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:44.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:44.638+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:44.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:44.622+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:44.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:44.622+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:44.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:44.622+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:44.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:44.622+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:44.911+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:44.911+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:44.911+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:44.912+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:45.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:45.635+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:45.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:45.635+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:45.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:45.635+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:45.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:45.636+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:45.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:45.624+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:45.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:45.624+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:45.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:45.624+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:45.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:45.624+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:45.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:45.865+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:45.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:45.865+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:45.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:45.865+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:45.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:45.866+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:46.610 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:14:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:14:46.334+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 27951 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:14:46.860 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:46.649+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:46.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:46.649+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:46.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:46.649+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:46.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:46.650+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:46.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:46.607+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:46.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:46.607+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:46.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:46.607+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:46.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:46.608+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:47.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:46.858+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:46.858+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:46.858+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:46.859+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:47.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:47.652+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:47.652+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:47.652+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:47.653+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:47.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:47.609+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:47.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:47.609+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:47.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:47.609+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:47.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:47.610+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:47.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:47.856+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:47.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:47.856+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:47.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:47.856+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:47.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:47.856+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:48.632+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:48.632+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:48.632+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:48.633+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:48.820+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:48.820+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:48.820+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:48.820+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:48.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:48.629+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:48.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:48.629+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:48.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:48.629+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:48.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:48.630+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:49.586+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:49.586+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:49.586+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:49.587+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:49.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:49.603+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:49.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:49.603+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:49.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:49.603+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:49.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:49.603+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:50.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:49.862+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:50.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:49.862+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:50.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:49.862+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:50.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:49.863+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:50.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:50.587+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:50.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:50.587+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:50.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:50.587+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:50.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:50.587+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:50.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:50.615+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:50.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:50.615+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:50.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:50.615+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:50.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:50.616+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:51.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:50.870+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:51.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:50.870+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:51.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:50.870+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:51.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:50.870+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:51.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:14:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:14:51.337+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 28043 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:14:51.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:51.578+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:51.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:51.578+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:51.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:51.578+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:51.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:51.578+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:51.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:51.573+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:51.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:51.573+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:51.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:51.573+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:51.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:51.574+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:51.845+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:51.845+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:51.845+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:51.845+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:52.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:52.614+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:52.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:52.614+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:52.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:52.614+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:52.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:52.614+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:52.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:52.532+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:52.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:52.532+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:52.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:52.532+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:52.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:52.533+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:53.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:52.878+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:52.878+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:52.878+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:52.879+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:53.860 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:53.604+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:53.860 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:53.604+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:53.860 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:53.604+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:53.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:53.605+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:53.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:53.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:53.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:53.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:53.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:53.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:53.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:53.580+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:53.857+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:53.857+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:53.857+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:54.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:53.858+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:54.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:54.621+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:54.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:54.621+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:54.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:54.621+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:54.885 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:54.622+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:54.885 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:54.564+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:54.885 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:54.564+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:54.885 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:54.564+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:54.885 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:54.564+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:54.881+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:54.881+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:54.881+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:54.882+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:55.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:55.653+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:55.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:55.653+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:55.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:55.653+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:55.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:55.653+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:55.912 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:55.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:55.912 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:55.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:55.912 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:55.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:55.912 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:55.587+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:56.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:55.908+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:55.908+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:55.908+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:55.908+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:56.694 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:14:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:14:56.341+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 28131 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:14:56.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:56.555+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:56.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:56.555+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:56.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:56.555+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:56.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:56.556+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:56.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:56.692+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:56.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:56.692+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:56.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:56.692+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:56.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:56.692+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:56.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:56.867+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:56.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:56.867+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:56.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:56.867+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:56.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:56.868+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:57.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:57.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:57.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:57.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:57.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:57.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:57.850 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:57.741+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:57.850 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:57.565+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:57.850 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:57.565+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:57.850 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:57.565+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:57.850 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:57.566+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:57.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:57.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:57.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:57.847+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:58.838 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:58.760+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:58.838 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:58.760+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:58.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:58.760+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:58.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:58.761+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:14:58.839 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:58.559+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:58.839 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:58.559+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:58.839 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:58.559+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:58.839 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:58.560+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:59.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:58.836+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:58.836+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:58.836+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:14:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:58.837+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:59.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:59.546+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:59.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:59.546+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:59.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:59.546+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:14:59.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:14:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:14:59.547+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:14:59.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:59.765+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:59.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:59.765+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:59.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:59.765+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:14:59.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:14:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:14:59.765+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:00.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:59.858+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:00.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:59.858+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:00.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:59.858+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:00.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:14:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:14:59.859+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:00.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:00.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:00.799+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:00.800+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:00.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:00.828+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:00.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:00.828+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:00.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:00.828+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:00.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:00.828+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:00.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:00.591+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:00.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:00.591+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:00.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:00.591+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:00.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:00.591+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:01.635 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:15:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:15:01.342+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 28222 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:15:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:01.814+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:01.814+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:01.814+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:01.815+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:01.631+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:01.631+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:01.631+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:01.632+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:01.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:01.788+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:01.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:01.788+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:01.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:01.788+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:01.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:01.789+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:02.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:02.778+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:02.850 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:02.778+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:02.850 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:02.778+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:02.850 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:02.779+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:02.850 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:02.583+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:02.850 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:02.583+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:02.851 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:02.583+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:02.851 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:02.583+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:02.846+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:02.846+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:02.846+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:02.846+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:03.826 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:03.767+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:03.826 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:03.767+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:03.826 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:03.767+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:03.826 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:03.768+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:03.826 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:03.539+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:03.827 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:03.539+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:03.827 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:03.539+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:03.827 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:03.539+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:03.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:03.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:03.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:03.823+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:04.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:04.747+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:04.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:04.747+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:04.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:04.747+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:04.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:04.747+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:04.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:04.776+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:04.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:04.776+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:04.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:04.776+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:04.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:04.776+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:04.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:04.564+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:04.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:04.564+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:04.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:04.564+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:04.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:04.565+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:05.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:05.702+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:05.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:05.702+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:05.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:05.702+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:05.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:05.703+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:05.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:05.802+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:05.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:05.802+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:05.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:05.802+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:05.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:05.803+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:05.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:05.567+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:05.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:05.568+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:05.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:05.568+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:05.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:05.568+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:06.600 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:15:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:15:06.347+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 28310 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:15:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:06.705+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:06.705+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:06.705+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:06.706+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:06.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:06.770+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:06.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:06.770+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:06.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:06.770+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:06.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:06.770+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:06.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:06.597+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:06.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:06.597+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:06.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:06.597+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:06.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:06.598+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:07.733+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:07.733+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:07.733+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:07.734+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:07.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:07.752+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:07.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:07.752+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:07.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:07.752+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:07.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:07.753+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:07.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:07.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:07.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:07.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:07.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:07.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:07.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:07.580+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:08.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:08.760+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:08.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:08.760+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:08.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:08.760+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:08.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:08.760+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:08.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:08.787+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:08.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:08.787+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:08.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:08.787+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:08.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:08.787+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:08.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:08.549+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:08.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:08.549+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:08.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:08.549+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:08.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:08.550+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:09.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:09.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:09.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:09.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:09.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:09.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:09.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:09.752+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:09.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:09.787+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:09.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:09.787+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:09.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:09.787+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:09.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:09.788+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:09.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:09.539+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:09.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:09.539+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:09.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:09.539+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:09.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:09.540+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:10.781+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:10.781+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:10.781+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:10.782+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:10.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:10.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:10.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:10.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:10.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:10.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:10.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:10.572+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:10.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:10.786+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:10.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:10.786+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:10.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:10.786+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:10.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:10.787+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:11.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:15:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:15:11.350+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 28401 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:15:11.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:11.525+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:11.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:11.525+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:11.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:11.525+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:11.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:11.526+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:11.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:11.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:11.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:11.797+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:11.757+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:12.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:11.757+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:12.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:11.757+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:12.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:11.758+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:12.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:12.755+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:12.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:12.755+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:12.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:12.755+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:12.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:12.755+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:12.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:12.734+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:12.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:12.734+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:12.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:12.734+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:12.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:12.735+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:12.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:12.529+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:12.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:12.529+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:12.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:12.529+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:12.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:12.530+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:13.715+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:13.715+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:13.715+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:13.716+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:13.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:13.736+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:13.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:13.736+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:13.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:13.736+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:13.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:13.737+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:13.511+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:13.511+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:13.511+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:13.512+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:14.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:14.716+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:14.719 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:14.716+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:14.719 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:14.716+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:14.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:14.466+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:14.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:14.466+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:14.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:14.466+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:14.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:14.467+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:15.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:14.717+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:14.720+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:14.720+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:14.720+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:14.721+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:15.450+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:15.450+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:15.450+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:15.451+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:16.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:15.729+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:16.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:15.729+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:16.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:15.729+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:16.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:15.730+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:15.739+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:16.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:15.739+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:16.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:15.739+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:16.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:15.740+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:16.709 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:15:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:15:16.354+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 28488 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:15:16.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:16.421+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:16.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:16.421+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:16.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:16.421+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:16.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:16.422+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:16.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:16.714+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:16.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:16.714+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:16.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:16.714+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:16.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:16.714+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:16.706+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:16.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:16.706+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:16.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:16.706+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:16.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:16.707+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:17.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:17.377+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:17.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:17.377+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:17.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:17.377+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:17.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:17.377+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:17.698+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:17.698+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:17.698+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:17.698+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:17.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:17.737+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:17.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:17.737+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:17.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:17.737+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:17.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:17.738+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:18.409+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:18.409+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:18.409+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:18.410+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:18.731+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:18.731+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:18.731+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:18.732+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:18.759+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:18.759+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:19.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:18.759+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:19.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:18.760+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:19.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:19.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:19.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:19.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:19.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:19.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:19.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:19.385+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:19.772+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:19.772+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:19.772+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:19.773+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:19.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:19.774+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:19.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:19.774+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:19.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:19.774+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:19.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:19.774+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:20.389+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:20.389+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:20.389+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:20.390+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:21.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:20.730+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:21.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:20.730+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:21.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:20.730+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:21.065 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:20.731+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:21.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:20.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:21.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:20.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:21.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:20.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:21.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:20.791+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:21.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:15:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:15:21.357+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 28579 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:15:21.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:21.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:21.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:21.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:21.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:21.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:21.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:21.372+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:21.819+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:21.819+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:21.819+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:21.820+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:21.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:22.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:21.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:22.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:21.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:22.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:21.723+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:22.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:22.391+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:22.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:22.391+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:22.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:22.391+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:22.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:22.392+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:22.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:22.707+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:22.707+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:22.707+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:22.708+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:22.849+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:22.849+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:22.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:22.849+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:22.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:22.850+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:23.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:23.356+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:23.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:23.356+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:23.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:23.356+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:23.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:23.357+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:23.887 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:23.752+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:23.888 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:23.752+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:23.888 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:23.752+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:23.888 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:23.752+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:24.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:23.885+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:24.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:23.885+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:24.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:23.885+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:24.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:23.886+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:24.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:24.314+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:24.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:24.315+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:24.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:24.315+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:24.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:24.315+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:25.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:24.898+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:25.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:24.898+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:25.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:24.898+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:25.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:24.899+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:25.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:24.730+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:25.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:24.730+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:25.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:24.730+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:25.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:24.731+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:25.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:25.312+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:25.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:25.312+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:25.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:25.312+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:25.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:25.312+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:25.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T20:15:25.602+0000 7fc14ad66640 0 monclient(hunting): authenticate timed out after 300 2024-09-20T20:15:25.607 INFO:teuthology.orchestra.run.smithi114.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-09-20T20:15:25.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:25.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:25.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:25.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:25.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:25.687+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:25.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:25.687+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:25.877+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:25.877+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:25.877+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:25.878+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:26.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T20:15:26.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -- ceph osd blacklist ls 2024-09-20T20:15:26.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:15:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:15:26.361+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 28666 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:15:26.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:26.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:26.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:26.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:26.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:26.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:26.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:26.337+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:26.717 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/mon.smithi114/config 2024-09-20T20:15:26.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:26.843+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:26.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:26.843+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:26.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:26.843+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:26.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:26.844+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:26.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:26.717+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:26.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:26.717+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:26.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:26.717+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:26.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:26.717+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:27.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:27.381+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:27.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:27.381+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:27.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:27.381+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:27.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:27.382+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:27.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:27.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:27.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:27.835+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:27.761+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:27.761+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:27.761+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:27.762+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:28.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:28.369+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:28.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:28.369+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:28.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:28.369+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:28.374 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:28.369+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:29.108 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:28.786+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:29.108 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:28.786+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:29.108 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:28.786+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:29.108 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:28.787+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:29.109 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:28.791+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:29.109 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:28.791+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:29.109 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:28.791+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:29.109 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:28.792+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:29.321+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:29.321+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:29.321+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:29.322+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:30.052 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:29.742+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:30.052 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:29.742+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:30.052 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:29.742+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:30.052 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:29.742+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:30.052 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:29.838+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:30.053 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:29.838+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:30.053 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:29.838+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:30.053 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:29.839+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:30.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:30.302+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:30.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:30.302+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:30.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:30.302+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:30.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:30.303+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:31.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:30.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:31.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:30.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:31.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:30.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:31.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:30.797+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:31.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:30.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:31.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:30.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:31.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:30.722+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:31.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:30.723+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:31.700 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:15:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:15:31.364+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 28758 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:15:31.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:31.275+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:31.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:31.275+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:31.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:31.275+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:31.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:31.276+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:31.843+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:31.843+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:31.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:31.843+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:31.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:31.843+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:31.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:31.697+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:31.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:31.697+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:31.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:31.697+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:31.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:31.698+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:32.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:32.308+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:32.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:32.308+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:32.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:32.308+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:32.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:32.308+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:32.866+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:32.866+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:32.866+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:32.867+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:32.734+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:33.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:32.734+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:33.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:32.734+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:33.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:32.735+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:33.631 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:33.329+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:33.631 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:33.329+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:33.631 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:33.329+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:33.631 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:33.330+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:33.857+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:33.857+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:33.857+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:33.858+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:33.699+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:33.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:33.699+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:33.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:33.699+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:33.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:33.700+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:34.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:34.286+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:34.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:34.286+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:34.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:34.286+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:34.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:34.287+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:34.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:34.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:34.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:34.873+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:34.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:34.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:34.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:34.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:34.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:34.671+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:34.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:34.672+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:35.326+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:35.326+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:35.326+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:35.327+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:35.824+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:35.824+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:35.824+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:35.825+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:35.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:35.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:35.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:35.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:35.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:35.650+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:35.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:35.650+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:36.650 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:15:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:15:36.368+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 28845 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:15:36.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:36.345+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:36.651 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:36.345+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:36.651 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:36.345+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:36.651 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:36.346+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:36.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:36.801+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:37.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:36.801+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:37.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:36.801+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:37.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:36.801+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:37.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:36.648+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:37.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:36.648+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:37.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:36.648+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:37.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:36.649+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:37.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:37.332+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:37.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:37.332+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:37.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:37.332+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:37.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:37.332+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:37.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:37.692+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:37.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:37.692+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:37.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:37.692+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:37.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:37.692+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:37.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:37.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:37.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:37.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:37.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:37.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:37.780+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:38.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:38.342+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:38.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:38.342+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:38.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:38.342+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:38.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:38.343+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:38.663+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:38.663+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:38.663+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:38.664+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:38.741+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:38.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:38.741+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:38.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:38.741+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:38.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:38.742+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:39.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:39.388+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:39.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:39.388+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:39.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:39.388+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:39.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:39.389+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:39.701+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:39.701+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:39.701+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:39.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:39.702+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:39.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:39.772+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:39.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:39.772+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:39.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:39.772+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:39.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:39.773+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:40.432+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:40.432+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:40.432+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:40.433+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:41.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:40.744+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:41.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:40.744+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:41.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:40.744+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:41.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:40.744+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:40.737+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:41.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:40.737+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:41.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:40.737+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:41.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:40.737+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:41.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:15:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:15:41.371+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 28936 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:15:41.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:41.471+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:41.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:41.471+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:41.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:41.471+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:41.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:41.471+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:42.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:41.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:42.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:41.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:42.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:41.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:42.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:41.733+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:41.772+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:42.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:41.772+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:42.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:41.772+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:42.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:41.772+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:42.731 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:42.476+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:42.731 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:42.476+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:42.731 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:42.476+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:42.731 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:42.477+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:42.732 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:42.696+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:42.732 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:42.696+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:42.732 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:42.696+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:42.732 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:42.697+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:42.728+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:42.728+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:42.728+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:42.729+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:43.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:43.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:43.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:43.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:43.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:43.736+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:43.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:43.737+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:43.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:43.724+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:43.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:43.724+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:43.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:43.724+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:43.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:43.725+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:43.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:43.514+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:43.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:43.514+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:43.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:43.514+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:43.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:43.514+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:44.746 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:44.704+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:44.747 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:44.704+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:44.747 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:44.704+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:44.747 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:44.704+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:44.747 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:44.473+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:44.747 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:44.473+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:44.747 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:44.473+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:44.748 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:44.474+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:45.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:44.744+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:44.744+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:44.744+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:44.745+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:45.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:45.471+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:45.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:45.471+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:45.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:45.471+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:45.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:45.472+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:45.751+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:45.751+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:45.751+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:45.752+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:45.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:45.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:45.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:45.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:45.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:45.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:45.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:45.667+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:46.711 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:15:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:15:46.374+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 29024 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:15:46.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:46.446+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:46.712 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:46.446+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:46.712 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:46.446+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:46.712 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:46.447+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:46.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:46.708+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:46.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:46.708+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:46.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:46.708+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:46.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:46.709+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:46.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:46.745+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:46.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:46.745+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:46.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:46.745+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:46.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:46.746+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:47.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:47.698+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:47.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:47.698+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:47.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:47.698+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:47.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:47.699+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:47.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:47.491+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:47.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:47.491+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:47.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:47.491+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:47.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:47.492+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:47.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:47.715+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:47.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:47.715+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:47.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:47.715+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:47.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:47.715+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:48.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:48.730+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:48.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:48.730+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:48.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:48.730+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:48.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:48.731+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:48.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:48.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:48.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:48.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:48.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:48.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:48.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:48.728+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:48.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:48.530+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:48.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:48.530+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:48.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:48.530+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:48.792 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:48.531+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:49.687+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:49.687+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:49.687+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:49.687+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:49.749+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:49.749+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:49.749+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:49.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:49.750+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:49.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:49.563+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:49.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:49.563+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:49.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:49.563+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:49.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:49.564+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:50.740+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:50.740+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:50.740+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:50.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:50.740+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:50.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:50.598+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:50.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:50.598+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:50.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:50.598+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:50.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:50.598+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:50.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:50.662+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:50.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:50.662+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:50.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:50.662+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:50.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:50.662+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:51.702 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:15:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:15:51.378+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 29116 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:15:51.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:51.623+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:51.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:51.623+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:51.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:51.623+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:51.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:51.624+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:51.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:51.560+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:51.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:51.560+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:51.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:51.560+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:51.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:51.561+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:51.699+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:51.699+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:51.699+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:51.700+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:52.578+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:52.578+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:52.578+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:52.579+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:52.656+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:52.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:52.656+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:52.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:52.656+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:52.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:52.657+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:52.577+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:52.577+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:52.577+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:52.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:52.578+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:53.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:53.653+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:53.653+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:53.653+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:53.653+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:53.562+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:53.562+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:53.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:53.562+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:53.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:53.563+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:53.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:53.590+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:53.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:53.590+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:53.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:53.590+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:53.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:53.591+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:54.589+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:54.589+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:54.589+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:54.590+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:54.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:54.685+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:54.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:54.685+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:54.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:54.685+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:54.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:54.686+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:54.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:54.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:54.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:54.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:54.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:54.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:54.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:54.580+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:55.867 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:55.558+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:55.867 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:55.558+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:55.867 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:55.558+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:55.867 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:55.559+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:55.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:55.708+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:55.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:55.708+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:55.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:55.708+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:55.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:55.708+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:55.868 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:55.613+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:55.868 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:55.613+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:55.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:55.613+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:55.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:55.614+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:56.694 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:15:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:15:56.382+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 29204 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:15:56.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:56.511+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:56.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:56.511+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:56.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:56.511+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:56.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:56.512+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:56.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:56.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:56.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:56.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:56.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:56.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:56.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:56.577+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:56.692+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:56.692+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:56.692+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:56.693+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:57.528+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:57.528+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:57.528+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:57.529+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:57.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:57.536+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:57.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:57.536+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:57.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:57.536+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:57.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:57.536+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:57.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:57.655+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:57.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:57.655+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:57.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:57.655+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:57.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:57.656+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:58.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:58.569+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:58.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:58.569+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:58.860 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:58.569+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:58.860 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:58.570+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:58.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:58.682+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:58.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:58.682+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:58.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:58.682+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:58.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:58.682+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:58.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:58.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:58.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:58.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:58.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:58.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:58.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:58.576+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:59.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:59.641+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:59.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:59.641+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:59.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:59.641+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:15:59.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:15:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:15:59.641+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:15:59.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:59.533+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:59.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:59.533+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:59.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:59.533+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:15:59.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:15:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:15:59.534+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:15:59.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:59.558+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:59.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:59.558+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:59.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:59.558+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:15:59.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:15:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:15:59.559+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:00.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:00.543+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:00.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:00.543+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:00.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:00.543+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:00.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:00.544+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:00.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:00.563+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:00.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:00.563+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:00.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:00.563+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:00.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:00.563+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:00.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:00.600+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:00.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:00.600+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:00.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:00.600+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:00.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:00.601+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:01.680 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:16:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:16:01.385+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 29295 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:16:01.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:01.498+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:01.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:01.498+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:01.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:01.498+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:01.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:01.498+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:01.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:01.596+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:01.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:01.596+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:01.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:01.596+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:01.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:01.596+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:01.682 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:01.583+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:01.682 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:01.583+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:01.682 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:01.583+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:01.683 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:01.584+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:02.584+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:02.584+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:02.584+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:02.585+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:02.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:02.622+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:02.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:02.622+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:02.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:02.622+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:02.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:02.623+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:02.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:02.489+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:02.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:02.489+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:02.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:02.489+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:02.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:02.490+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:03.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:03.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:03.660+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:03.660+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:03.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:03.537+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:03.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:03.537+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:03.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:03.537+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:03.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:03.538+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:03.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:03.507+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:03.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:03.507+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:03.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:03.507+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:03.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:03.508+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:04.752 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:04.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:04.752 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:04.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:04.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:04.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:04.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:04.644+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:04.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:04.488+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:04.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:04.488+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:04.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:04.488+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:04.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:04.489+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:04.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:04.487+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:04.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:04.487+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:04.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:04.487+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:04.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:04.487+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:05.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:05.501+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:05.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:05.501+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:05.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:05.501+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:05.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:05.502+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:05.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:05.517+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:05.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:05.517+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:05.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:05.517+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:05.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:05.518+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:05.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:05.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:05.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:05.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:05.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:05.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:05.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:05.620+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:06.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:16:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:16:06.389+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 29382 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:16:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:06.528+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:06.528+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:06.528+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:06.529+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:06.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:06.521+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:06.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:06.521+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:06.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:06.521+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:06.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:06.521+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:06.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:06.592+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:06.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:06.592+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:06.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:06.592+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:06.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:06.593+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:07.551+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:07.551+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:07.551+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:07.552+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:07.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:07.488+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:07.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:07.488+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:07.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:07.488+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:07.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:07.489+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:07.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:07.547+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:07.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:07.547+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:07.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:07.547+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:07.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:07.547+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:08.524+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:08.524+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:08.524+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:08.525+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:08.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:08.549+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:08.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:08.549+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:08.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:08.549+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:08.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:08.550+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:08.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:08.560+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:08.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:08.560+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:08.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:08.560+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:08.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:08.561+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:09.504+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:09.504+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:09.504+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:09.505+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:09.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:09.547+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:09.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:09.547+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:09.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:09.547+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:09.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:09.548+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:09.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:09.548+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:09.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:09.548+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:09.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:09.548+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:09.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:09.549+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:10.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:10.457+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:10.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:10.457+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:10.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:10.457+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:10.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:10.457+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:10.897 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:10.576+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:10.898 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:10.576+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:10.898 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:10.576+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:10.898 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:10.576+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:10.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:10.593+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:10.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:10.593+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:10.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:10.593+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:10.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:10.594+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:11.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:16:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:16:11.392+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 29474 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:16:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:11.633+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:11.633+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:11.633+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:11.634+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:11.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:11.436+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:11.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:11.436+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:11.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:11.436+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:11.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:11.436+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:11.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:11.570+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:11.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:11.570+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:11.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:11.570+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:11.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:11.571+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:12.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:12.660+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:12.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:12.660+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:12.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:12.660+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:12.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:12.661+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:12.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:12.537+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:12.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:12.537+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:12.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:12.537+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:12.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:12.538+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:12.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:12.486+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:12.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:12.486+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:12.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:12.486+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:12.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:12.487+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:13.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:13.461+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:13.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:13.461+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:13.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:13.461+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:13.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:13.461+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:13.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:13.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:13.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:13.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:13.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:13.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:13.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:13.673+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:13.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:13.504+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:13.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:13.504+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:13.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:13.504+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:13.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:13.505+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:14.456+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:14.456+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:14.456+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:14.457+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:14.644+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:14.644+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:14.644+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:14.645+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:14.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:14.499+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:14.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:14.499+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:14.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:14.499+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:14.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:14.500+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:15.430+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:15.430+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:15.430+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:15.430+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:15.657+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:15.657+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:15.657+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:15.657+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:15.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:15.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:15.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:15.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:15.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:15.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:15.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:15.490+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:16.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:16:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:16:16.396+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 29562 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:16:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:16.637+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:16.637+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:16.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:16.637+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:16.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:16.638+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:16.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:16.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:16.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:16.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:16.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:16.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:16.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:16.515+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:16.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:16.440+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:16.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:16.440+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:16.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:16.440+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:16.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:16.441+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:17.481 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:17.418+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:17.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:17.418+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:17.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:17.418+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:17.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:17.419+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:17.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:17.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:17.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:17.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:17.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:17.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:17.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:17.668+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:17.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:17.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:17.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:17.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:17.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:17.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:17.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:17.480+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:18.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:18.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:18.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:18.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:18.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:18.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:18.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:18.491+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:18.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:18.370+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:18.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:18.370+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:18.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:18.370+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:18.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:18.371+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:18.625+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:18.625+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:18.625+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:18.626+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:19.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:19.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:19.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:19.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:19.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:19.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:19.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:19.524+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:19.609 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:19.330+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:19.609 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:19.330+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:19.609 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:19.330+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:19.609 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:19.331+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:19.606+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:19.606+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:19.606+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:19.606+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:20.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:20.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:20.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:46.104162+0000 (oldest deadline 2024-09-20T20:06:06.104162+0000) 2024-09-20T20:16:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:20.353+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:20.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:20.601+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:20.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:20.601+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:20.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:20.601+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:20.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:20.602+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:20.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:20.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:20.937 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:20.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:20.937 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:20.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:20.937 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:20.524+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:21.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:16:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:16:21.399+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 29652 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:16:21.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:21.337+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:21.571+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:21.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:21.571+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:21.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:21.571+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:05:51.165852+0000 (oldest deadline 2024-09-20T20:06:11.165852+0000) 2024-09-20T20:16:21.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:21.572+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:21.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:21.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:21.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:21.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:21.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:21.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:21.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:21.504+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:22.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:22.293+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:22.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:22.543+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:22.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:22.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:22.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:22.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:22.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:22.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:22.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:22.511+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:23.290+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:23.507+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:23.494+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:23.494+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:23.494+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:23.495+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:24.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:24.319+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:24.480+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:24.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:24.496+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:24.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:24.496+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:24.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:24.496+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:24.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:24.497+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:25.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:25.298+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:25.452+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:25.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:25.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:25.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:25.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:25.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:25.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:25.545+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:26.503+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:26.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:16:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:16:26.403+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 29740 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:16:26.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:26.347+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:26.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:26.583+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:26.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:26.583+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:26.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:26.583+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:26.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:26.583+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:27.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:27.485+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:27.581 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:27.299+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:27.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:27.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:27.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:27.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:27.578+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:28.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:28.470+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:28.584 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:28.322+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:28.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:28.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:28.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:28.582+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:29.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:29.509+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:29.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:29.344+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:29.539+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:29.539+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:29.539+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:29.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:29.540+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:30.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:30.393+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:30.503+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:30.567+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:30.567+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:30.567+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:30.568+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:31.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:31.352+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:31.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:16:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:16:31.407+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 29832 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:16:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:31.469+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:31.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:31.571+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:31.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:31.571+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:31.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:31.571+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:31.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:31.571+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:32.421+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:32.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:32.365+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:32.538+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:32.538+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:32.538+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:32.538+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:33.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:33.400+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:33.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:33.399+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:33.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:33.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:33.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:33.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:33.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:33.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:33.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:33.504+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:34.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:34.391+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:34.381+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:34.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:34.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:34.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:34.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:34.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:34.544+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:35.403+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:35.419+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:35.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:35.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:35.587+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:06:03.684146+0000 (oldest deadline 2024-09-20T20:06:23.684146+0000) 2024-09-20T20:16:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:35.588+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:36.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:16:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:16:36.409+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 29920 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:16:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:36.593+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:36.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:36.401+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:36.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:36.441+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:37.616+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:37.430+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:37.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:37.407+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:38.629+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:38.471+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:38.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:38.405+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:39.677 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:39.489+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:39.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:39.417+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:39.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:39.674+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:40.422+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:40.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:40.698+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:40.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:40.470+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:41.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:41.397+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:41.492 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:16:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:16:41.414+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 30011 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:16:41.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:41.717+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:41.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:41.489+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:42.448+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:42.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:42.362+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:43.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:42.753+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:43.635 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:43.437+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:43.635 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:43.437+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:43.635 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:43.437+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:43.635 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:43.438+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:43.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:43.356+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:43.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:43.356+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:43.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:43.356+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:43.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:43.357+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:43.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:43.803+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:44.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:44.388+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:44.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:44.388+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:44.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:44.388+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:44.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:44.388+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:44.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:44.324+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:44.690 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:44.324+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:44.690 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:44.324+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:44.690 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:44.325+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:44.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:44.851+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:45.289+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:45.289+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:45.289+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:45.290+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:45.371+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:45.371+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:45.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:45.371+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:45.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:45.372+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:46.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:45.812+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:46.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:16:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:16:46.418+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 30099 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:16:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:46.415+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:46.415+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:46.415+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:46.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:46.416+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:46.298+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:46.298+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:46.298+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:46.299+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:47.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:46.846+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:47.374+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:47.374+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:47.374+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:47.374+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:47.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:47.267+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:47.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:47.267+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:47.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:47.267+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:47.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:47.268+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:48.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:47.817+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:48.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:48.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:48.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:48.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:48.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:48.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:48.341+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:48.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:48.219+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:48.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:48.219+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:48.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:48.219+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:48.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:48.220+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:48.807+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:49.336+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:49.336+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:49.336+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:49.337+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:49.245+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:49.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:49.245+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:49.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:49.245+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:49.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:49.246+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:50.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:49.857+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:50.383+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:50.383+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:50.383+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:50.384+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:50.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:50.294+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:50.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:50.294+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:50.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:50.294+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:50.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:50.295+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:51.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:50.818+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:51.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:16:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:16:51.422+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 30191 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:16:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:51.400+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:51.400+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:51.400+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:51.401+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:51.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:51.269+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:51.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:51.269+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:51.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:51.269+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:51.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:51.270+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:51.828+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:52.450+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:52.450+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:52.450+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:52.451+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:52.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:52.313+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:52.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:52.313+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:52.313+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:52.313+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:52.862+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:53.635 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:53.410+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:53.635 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:53.410+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:53.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:53.410+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:53.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:53.411+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:53.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:53.300+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:53.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:53.300+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:53.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:53.300+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:53.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:53.301+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:53.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:53.864+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:54.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:54.267+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:54.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:54.267+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:54.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:54.267+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:54.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:54.267+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:54.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:54.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:54.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:54.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:54.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:54.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:54.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:54.383+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:55.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:54.883+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:55.246+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:55.246+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:55.246+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:55.247+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:55.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:55.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:55.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:55.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:55.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:55.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:55.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:55.392+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:55.852+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:56.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:16:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:16:56.426+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 30279 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:16:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:56.394+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:56.394+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:56.394+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:56.395+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:56.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:56.290+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:56.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:56.290+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:56.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:56.290+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:56.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:56.291+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:56.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:16:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:56.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:16:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:56.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:16:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:56.891+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:57.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:57.334+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:57.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:57.334+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:57.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:57.334+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:57.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:57.335+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:57.399+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:57.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:57.399+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:57.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:57.399+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:57.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:57.399+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:57.897+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:16:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:57.897+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:16:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:57.897+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:16:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:57.897+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:58.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:58.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:58.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:58.382+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:58.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:58.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:58.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:58.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:58.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:58.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:58.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:58.372+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:58.914+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:16:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:58.914+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:16:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:58.914+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:16:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:58.915+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:16:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:59.398+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:59.398+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:59.398+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:16:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:16:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:16:59.399+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:16:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:59.365+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:59.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:59.365+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:59.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:59.365+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:16:59.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:16:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:16:59.366+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:00.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:59.943+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:00.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:59.943+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:00.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:59.943+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:00.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:16:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:16:59.943+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:00.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:00.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:00.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:00.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:00.382+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:00.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:00.347+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:00.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:00.347+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:00.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:00.347+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:00.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:00.348+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:01.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:00.930+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:01.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:00.930+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:01.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:00.930+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:01.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:00.930+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:01.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:17:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:17:01.430+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 30370 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:17:01.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:01.391+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:01.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:01.391+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:01.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:01.391+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:01.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:01.392+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:01.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:01.355+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:01.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:01.355+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:01.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:01.355+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:01.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:01.355+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:01.909+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:01.909+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:01.909+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:01.910+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:02.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:02.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:02.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:02.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:02.341+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:02.383+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:02.383+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:02.383+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:02.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:02.384+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:02.950+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:02.950+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:02.950+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:02.951+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:03.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:03.352+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:03.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:03.352+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:03.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:03.352+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:03.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:03.353+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:03.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:03.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:03.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:03.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:03.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:03.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:03.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:03.359+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:03.946+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:03.946+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:03.946+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:04.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:03.946+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:04.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:04.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:04.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:04.333+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:04.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:04.322+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:04.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:04.322+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:04.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:04.322+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:04.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:04.323+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:04.945+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:04.945+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:04.945+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:04.946+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:05.329+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:05.329+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:05.329+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:05.330+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:05.303+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:05.303+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:05.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:05.303+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:05.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:05.304+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:06.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:05.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:06.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:05.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:06.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:05.896+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:06.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:05.897+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:06.447 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:17:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:17:06.433+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 30458 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:17:06.447 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:06.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:06.447 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:06.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:06.448 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:06.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:06.448 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:06.330+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:06.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:06.328+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:06.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:06.328+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:06.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:06.328+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:06.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:06.329+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:07.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:06.854+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:07.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:06.854+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:07.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:06.854+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:07.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:06.855+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:07.323+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:07.323+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:07.323+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:07.323+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:07.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:07.306+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:07.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:07.306+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:07.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:07.306+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:07.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:07.307+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:07.825+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:07.825+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:07.825+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:07.826+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:08.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:08.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:08.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:08.335+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:08.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:08.325+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:08.325+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:08.325+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:08.326+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:09.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:08.862+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:09.243 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:08.862+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:09.243 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:08.862+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:09.243 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:08.862+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:09.321+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:09.321+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:09.321+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:09.321+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:09.290+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:09.290+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:09.290+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:09.291+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:09.844+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:09.844+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:09.844+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:09.845+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:10.313+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:10.313+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:10.313+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:10.313+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:10.247+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:10.247+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:10.247+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:10.247+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:11.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:10.853+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:11.211 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:10.853+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:11.211 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:10.853+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:11.211 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:10.853+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:11.211 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:11.208+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:11.211 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:11.208+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:11.212 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:11.208+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:11.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:17:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:17:11.437+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 30550 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:17:11.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:11.209+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:11.358+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:11.358+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:11.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:11.358+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:11.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:11.359+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:11.821+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:11.821+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:11.821+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:11.822+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:12.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:12.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:12.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:12.233+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:12.349+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:12.349+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:12.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:12.349+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:12.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:12.350+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:13.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:12.837+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:12.837+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:12.837+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:13.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:12.838+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:13.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:13.313+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:13.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:13.313+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:13.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:13.313+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:13.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:13.313+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:13.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:13.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:13.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:13.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:13.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:13.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:13.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:13.230+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:13.817+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:13.817+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:13.817+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:13.817+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:14.359+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:14.359+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:14.359+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:14.360+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:14.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:14.209+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:14.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:14.209+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:14.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:14.209+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:14.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:14.210+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:15.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:14.808+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:15.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:14.808+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:15.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:14.808+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:15.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:14.808+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:15.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:15.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:15.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:15.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:15.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:15.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:15.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:15.379+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:15.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:15.210+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:15.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:15.210+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:15.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:15.210+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:15.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:15.211+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:16.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:15.818+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:16.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:15.818+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:16.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:15.818+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:16.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:15.818+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:16.420 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:16.164+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:16.420 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:16.164+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:16.420 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:16.164+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:16.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:16.165+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:16.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:17:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:17:16.441+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 30638 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:17:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:16.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:16.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:16.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:16.419+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:17.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:16.799+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:17.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:16.799+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:17.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:16.799+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:17.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:16.800+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:17.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:17.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:17.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:17.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:17.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:17.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:17.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:17.132+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:17.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:17.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:17.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:17.436+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:18.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:17.777+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:18.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:17.777+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:18.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:17.777+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:18.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:17.778+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:18.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:18.114+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:18.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:18.114+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:18.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:18.114+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:18.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:18.115+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:18.797 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:18.483+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:18.797 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:18.483+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:18.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:18.483+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:18.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:18.484+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:19.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:18.793+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:19.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:18.793+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:19.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:18.793+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:19.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:18.794+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:19.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:19.066+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:19.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:19.066+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:19.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:19.066+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:19.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:19.067+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:19.453+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:19.453+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:19.453+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:19.454+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:20.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:19.772+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:20.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:19.772+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:20.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:19.772+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:20.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:19.773+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:20.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:20.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:20.411 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:20.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:20.411 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:20.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:20.411 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:20.070+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:20.407+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:20.407+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:20.407+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:20.407+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:21.045 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:20.766+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:21.045 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:20.766+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:21.045 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:20.766+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:21.045 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:20.767+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:21.447 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:21.041+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:21.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:21.041+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:21.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:21.041+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:21.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:21.042+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:21.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:17:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:17:21.444+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 30729 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:17:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:21.457+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:21.457+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:21.457+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:21.458+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:22.026 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:21.738+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:22.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:21.738+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:22.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:21.738+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:22.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:21.738+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:22.442 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:22.024+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:22.442 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:22.024+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:22.442 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:22.024+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:22.442 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:22.024+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:22.439+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:22.439+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:22.439+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:22.439+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:23.071 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:22.786+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:23.071 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:22.786+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:23.071 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:22.786+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:23.071 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:22.787+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:23.414 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:23.068+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:23.414 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:23.068+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:23.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:23.068+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:23.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:23.069+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:23.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:23.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:23.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:23.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:23.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:23.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:23.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:23.412+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:24.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:23.825+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:24.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:23.825+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:24.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:23.825+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:24.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:23.826+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:24.079 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:24.075+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:24.079 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:24.075+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:24.079 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:24.075+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:24.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:24.076+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:24.624 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:24.369+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:24.624 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:24.369+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:24.624 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:24.369+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:24.625 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:24.370+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:24.828+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:24.828+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:24.828+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:24.828+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:25.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:25.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:25.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:25.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:25.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:25.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:25.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:25.077+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:25.380+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:25.380+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:25.380+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:25.381+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:26.065 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:25.797+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:26.065 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:25.797+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:26.065 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:25.797+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:26.065 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:25.798+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:26.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:26.062+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:26.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:26.062+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:26.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:26.062+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:26.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:26.063+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:26.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:17:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:17:26.449+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 30817 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:17:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:26.354+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:26.354+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:26.354+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:26.355+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:27.105 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:26.800+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:27.105 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:26.800+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:27.105 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:26.800+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:27.105 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:26.800+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:27.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:27.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:27.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:27.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:27.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:27.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:27.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:27.340+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:27.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:27.102+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:27.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:27.102+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:27.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:27.102+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:27.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:27.103+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:28.105 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:27.776+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:28.106 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:27.776+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:28.106 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:27.776+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:28.106 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:27.777+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:28.369 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:28.103+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:28.369 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:28.103+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:28.369 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:28.103+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:28.370 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:28.104+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:28.367+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:28.367+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:28.367+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:28.367+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:29.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:28.734+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:29.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:28.734+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:29.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:28.734+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:29.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:28.735+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:29.395 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:29.111+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:29.396 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:29.111+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:29.396 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:29.111+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:29.396 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:29.112+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:29.393+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:29.393+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:29.393+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:29.394+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:30.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:29.758+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:30.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:29.758+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:30.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:29.758+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:30.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:29.758+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:30.420 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:30.159+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:30.420 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:30.159+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:30.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:30.159+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:30.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:30.160+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:30.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:30.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:30.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:30.418+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:31.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:30.797+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:31.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:30.797+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:31.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:30.797+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:31.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:30.798+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:31.455 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:31.374+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:31.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:31.374+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:31.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:31.374+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:31.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:31.375+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:31.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:31.183+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:31.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:31.183+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:31.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:31.183+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:31.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:31.184+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:31.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:17:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:17:31.453+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 30909 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:17:32.035 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:31.836+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:32.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:31.836+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:32.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:31.836+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:32.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:31.837+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:32.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:32.150+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:32.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:32.150+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:32.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:32.150+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:32.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:32.151+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:32.399+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:32.399+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:32.399+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:32.400+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:33.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:32.800+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:33.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:32.800+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:33.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:32.800+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:33.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:32.801+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:33.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:33.159+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:33.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:33.159+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:33.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:33.159+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:33.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:33.160+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:33.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:33.365+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:33.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:33.365+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:33.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:33.365+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:33.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:33.366+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:33.810+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:33.810+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:33.810+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:33.811+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:34.153+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:34.153+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:34.153+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:34.153+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:34.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:34.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:34.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:34.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:34.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:34.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:34.346+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:35.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:34.817+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:35.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:34.817+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:35.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:34.817+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:35.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:34.817+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:35.172+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:35.172+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:35.172+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:35.173+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:35.364+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:35.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:35.364+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:35.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:35.364+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:35.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:35.364+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:36.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:35.824+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:36.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:35.824+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:36.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:35.824+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:36.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:35.825+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:36.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:36.127+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:36.460 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:36.127+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:36.460 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:36.127+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:36.460 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:36.127+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:36.460 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:36.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:36.460 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:36.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:36.460 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:36.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:36.460 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:36.319+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:36.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:17:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:17:36.457+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 30997 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:17:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:36.872+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:36.872+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:36.872+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:36.873+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:37.103+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:37.103+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:37.103+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:37.104+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:37.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:37.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:37.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:37.341+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:38.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:37.915+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:38.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:37.915+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:38.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:37.915+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:38.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:37.915+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:38.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:38.154+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:38.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:38.154+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:38.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:38.154+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:38.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:38.154+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:38.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:38.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:38.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:38.294+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:39.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:39.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:39.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:39.131+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:39.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:38.921+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:39.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:38.921+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:39.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:38.921+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:39.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:38.922+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:39.246+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:39.246+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:39.246+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:39.247+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:40.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:39.895+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:40.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:39.895+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:40.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:39.895+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:40.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:39.896+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:40.212+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:40.212+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:40.212+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:40.212+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:40.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:40.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:40.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:40.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:40.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:40.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:40.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:40.171+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:41.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:40.897+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:41.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:40.897+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:41.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:40.897+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:41.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:40.897+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:41.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:41.187+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:41.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:41.187+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:41.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:41.187+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:41.464 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:41.188+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:41.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:41.162+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:41.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:41.162+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:41.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:41.162+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:41.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:41.162+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:41.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:17:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:17:41.461+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 31088 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:17:42.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:41.918+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:42.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:41.918+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:42.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:41.918+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:42.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:41.919+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:42.166+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:42.166+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:42.166+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:42.167+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:42.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:42.126+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:42.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:42.126+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:42.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:42.126+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:42.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:42.127+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:43.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:42.963+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:43.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:42.963+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:43.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:42.963+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:43.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:42.964+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:43.152+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:43.152+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:43.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:43.152+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:43.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:43.152+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:43.638 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:43.217+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:43.638 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:43.217+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:43.638 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:43.217+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:43.638 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:43.218+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:43.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:43.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:43.934+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:43.935+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:44.140+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:44.140+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:44.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:44.140+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:44.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:44.140+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:44.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:44.251+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:44.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:44.251+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:44.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:44.251+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:44.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:44.252+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:45.096+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:45.096+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:45.096+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:45.096+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:44.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:45.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:44.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:45.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:44.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:45.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:44.926+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:45.293+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:45.293+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:45.293+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:45.294+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:46.147 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:45.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:46.148 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:45.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:46.148 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:45.887+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:46.148 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:45.887+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:46.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:46.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:46.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:46.285+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:46.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:46.145+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:46.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:46.145+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:46.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:46.145+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:46.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:46.146+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:46.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:17:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:17:46.465+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 31176 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:17:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:46.894+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:46.894+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:46.894+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:46.895+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:47.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:47.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:47.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:47.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:47.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:47.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:47.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:47.109+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:47.304+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:47.304+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:47.304+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:47.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:47.305+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:47.871+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:47.871+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:47.871+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:47.872+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:48.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:48.114+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:48.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:48.114+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:48.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:48.114+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:48.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:48.114+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:48.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:48.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:48.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:48.352+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:49.122 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:48.855+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:49.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:48.855+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:49.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:48.855+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:49.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:48.855+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:49.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:49.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:49.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:49.347+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:49.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:49.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:49.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:49.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:49.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:49.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:49.121+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:50.090 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:49.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:50.090 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:49.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:50.090 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:49.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:50.090 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:49.833+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:50.393 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:50.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:50.393 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:50.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:50.393 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:50.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:50.393 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:50.088+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:50.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:50.390+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:50.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:50.390+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:50.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:50.390+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:50.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:50.391+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:51.093 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:50.818+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:51.094 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:50.818+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:51.094 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:50.818+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:51.094 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:50.818+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:51.385 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:51.090+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:51.385 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:51.091+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:51.385 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:51.091+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:51.385 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:51.091+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:51.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:17:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:17:51.468+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 31268 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:17:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:51.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:51.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:51.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:51.383+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:52.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:51.856+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:52.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:51.856+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:52.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:51.856+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:52.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:51.856+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:52.353 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:52.053+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:52.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:52.053+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:52.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:52.053+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:52.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:52.054+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:52.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:52.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:52.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:52.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:52.352+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:53.085+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:53.085+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:53.085+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:53.086+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:53.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:52.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:53.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:52.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:53.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:52.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:53.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:52.848+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:53.639 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:53.352+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:53.639 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:53.352+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:53.639 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:53.352+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:53.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:53.353+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:53.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:53.883+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:53.883+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:53.883+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:53.884+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:54.068+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:54.068+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:54.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:54.068+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:54.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:54.069+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:54.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:54.309+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:54.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:54.309+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:54.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:54.309+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:54.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:54.309+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:54.891+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:54.891+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:54.891+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:55.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:54.892+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:55.020+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:55.020+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:55.020+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:55.020+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:55.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:55.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:55.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:55.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:55.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:55.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:55.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:55.278+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:56.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:56.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:56.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:56.060+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:55.895+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:56.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:55.895+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:56.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:55.895+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:56.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:55.896+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:56.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:17:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:17:56.472+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 31356 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:17:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:56.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:56.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:56.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:56.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:56.329+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:57.078+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:57.078+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:57.078+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:57.079+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:56.900+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:57.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:56.900+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:57.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:56.900+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:57.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:56.900+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:57.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:57.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:57.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:57.298+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:58.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:58.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:58.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:58.110+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:58.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:57.871+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:58.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:57.871+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:58.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:57.871+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:58.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:57.871+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:58.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:58.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:58.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:58.335+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:17:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:59.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:59.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:59.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:17:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:17:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:17:59.119+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:58.915+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:59.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:58.915+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:59.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:58.915+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:17:59.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:58.917+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:17:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:59.366+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:59.366+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:59.366+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:17:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:17:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:17:59.366+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:00.129+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:00.129+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:00.129+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:00.129+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:00.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:59.893+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:00.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:59.893+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:00.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:59.893+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:00.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:17:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:17:59.894+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:00.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:00.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:00.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:00.331+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:01.097 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:00.854+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:01.097 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:00.854+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:01.098 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:00.854+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:01.098 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:00.855+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:01.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:01.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:01.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:01.312+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:01.095+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:01.095+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:01.095+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:01.095+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:01.762 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:18:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:18:01.476+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 31447 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:18:02.069 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:01.863+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:02.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:01.863+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:02.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:01.863+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:02.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:01.864+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:02.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:02.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:02.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:02.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:02.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:02.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:02.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:02.295+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:02.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:02.067+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:02.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:02.067+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:02.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:02.067+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:02.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:02.068+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:02.865+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:02.865+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:02.865+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:02.865+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:03.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:03.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:03.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:03.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:03.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:03.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:03.082+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:03.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:03.337+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:03.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:03.337+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:03.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:03.337+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:03.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:03.338+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:03.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:03.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:03.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:03.877+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:04.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:04.034+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:04.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:04.034+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:04.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:04.034+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:04.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:04.035+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:04.370+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:04.370+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:04.370+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:04.370+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:04.895+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:04.895+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:04.895+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:04.896+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:05.024+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:05.024+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:05.024+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:05.024+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:05.403+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:05.403+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:05.403+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:05.403+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:06.071+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:06.071+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:06.071+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:06.072+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:05.904+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:06.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:05.904+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:06.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:05.904+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:06.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:05.905+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:06.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:18:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:18:06.479+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 31535 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:18:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:06.442+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:06.442+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:06.442+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:06.443+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:07.056+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:07.056+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:07.056+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:07.056+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:07.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:06.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:07.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:06.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:07.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:06.917+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:07.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:06.917+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:07.433+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:07.433+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:07.433+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:07.434+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:08.048+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:08.048+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:08.048+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:08.048+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:08.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:07.948+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:08.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:07.948+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:08.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:07.948+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:08.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:07.948+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:08.405+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:08.405+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:08.405+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:08.405+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:09.363 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:08.976+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:09.363 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:08.976+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:09.363 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:08.976+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:09.364 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:08.977+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:09.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:09.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:09.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:09.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:09.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:09.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:09.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:09.038+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:09.360+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:09.360+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:09.360+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:09.361+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:10.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:09.984+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:10.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:09.984+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:10.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:09.984+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:10.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:09.985+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:10.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:10.039+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:10.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:10.039+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:10.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:10.039+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:10.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:10.040+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:10.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:10.385+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:10.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:10.385+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:10.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:10.385+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:10.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:10.386+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:11.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:11.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:11.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:11.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:11.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:11.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:11.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:11.076+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:11.392 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:10.983+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:11.393 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:10.983+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:11.393 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:10.983+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:11.393 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:10.984+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:11.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:18:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:18:11.484+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 31627 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:18:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:11.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:11.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:11.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:11.390+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:12.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:12.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:12.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:12.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:12.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:12.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:12.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:12.003+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:12.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:12.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:12.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:12.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:12.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:12.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:12.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:12.111+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:12.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:12.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:12.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:12.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:12.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:12.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:12.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:12.390+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:13.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:13.008+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:13.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:13.008+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:13.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:13.008+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:13.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:13.008+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:13.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:13.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:13.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:13.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:13.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:13.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:13.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:13.077+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:13.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:13.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:13.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:13.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:13.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:13.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:13.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:13.363+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:14.348 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:14.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:14.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:14.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:14.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:14.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:14.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:14.013+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:14.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:14.127+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:14.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:14.127+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:14.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:14.127+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:14.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:14.127+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:14.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:14.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:14.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:14.346+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:15.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:15.063+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:15.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:15.063+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:15.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:15.063+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:15.348 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:15.064+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:15.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:15.147+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:15.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:15.147+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:15.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:15.147+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:15.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:15.147+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:15.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:15.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:15.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:15.345+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:16.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:16.095+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:16.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:16.095+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:16.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:16.095+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:16.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:16.096+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:16.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:16.140+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:16.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:16.140+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:16.374 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:16.140+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:16.374 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:16.141+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:16.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:18:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:18:16.489+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 31715 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:18:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:16.370+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:16.370+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:16.370+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:16.371+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:17.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:17.114+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:17.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:17.114+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:17.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:17.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:17.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:17.115+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:17.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:17.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:17.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:17.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:17.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:17.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:17.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:17.333+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:17.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:17.132+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:17.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:17.132+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:17.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:17.132+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:17.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:17.133+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:18.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:18.120+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:18.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:18.120+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:18.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:18.120+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:18.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:18.120+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:18.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:18.067+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:18.329 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:18.067+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:18.329 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:18.067+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:18.329 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:18.068+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:18.325+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:18.325+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:18.325+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:18.326+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:19.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:19.021+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:19.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:19.021+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:19.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:19.021+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:19.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:19.022+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:19.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:19.075+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:19.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:19.075+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:19.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:19.075+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:19.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:19.076+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:19.314+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:19.314+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:19.314+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:19.314+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:20.330 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:20.032+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:20.331 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:20.032+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:20.331 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:20.032+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:20.331 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:20.033+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:20.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:19.982+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:20.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:19.982+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:20.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:19.982+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:20.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:19.983+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:20.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:20.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:20.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:20.328+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:21.033 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:20.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:21.033 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:20.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:21.033 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:20.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:21.033 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:20.981+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:21.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:21.030+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:21.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:21.030+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:21.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:21.030+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:21.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:21.031+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:21.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:18:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:18:21.493+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 31803 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:18:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:21.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:21.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:21.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:21.345+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:22.026 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:21.999+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:22.026 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:21.999+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:22.026 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:21.999+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:22.026 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:22.000+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:22.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:22.023+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:22.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:22.023+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:22.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:22.023+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:22.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:22.024+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:22.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:22.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:22.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:22.346+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:23.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:23.005+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:23.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:23.005+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:23.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:23.005+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:23.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:23.006+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:23.065+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:23.065+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:23.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:23.065+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:23.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:23.065+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:23.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:23.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:23.642 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:23.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:23.642 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:23.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:23.642 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:23.336+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:24.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:24.042+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:24.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:24.042+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:24.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:24.042+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:24.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:24.042+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:24.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:24.061+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:24.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:24.061+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:24.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:24.061+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:24.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:24.062+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:24.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:24.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:24.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:24.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:24.384+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:25.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:25.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:25.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:25.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:25.406 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:25.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:25.406 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:25.003+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:25.406 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:25.067+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:25.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:25.067+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:25.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:25.067+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:25.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:25.068+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:25.402+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:25.402+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:25.402+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:25.403+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:26.395 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:26.019+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:26.395 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:26.019+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:26.395 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:26.019+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:26.395 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:26.019+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:26.395 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:26.090+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:26.395 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:26.090+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:26.396 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:26.090+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:26.396 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:26.091+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:26.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:18:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:18:26.497+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 31885 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:18:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:26.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:26.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:26.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:26.393+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:27.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:27.066+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:27.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:27.066+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:27.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:27.066+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:27.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:27.067+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:27.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:27.385+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:27.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:27.385+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:27.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:27.385+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:27.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:27.386+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:27.137+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:27.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:27.137+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:27.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:27.137+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:27.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:27.138+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:28.107+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:28.107+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:28.107+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:28.107+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:28.046+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:28.046+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:28.046+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:28.046+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:28.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:28.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:28.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:28.386+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:29.344 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:29.078+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:29.344 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:29.078+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:29.345 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:29.078+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:29.345 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:29.079+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:29.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:29.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:29.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:29.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:29.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:29.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:29.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:29.081+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:29.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:29.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:29.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:29.342+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:30.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:30.063+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:30.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:30.063+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:30.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:30.063+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:30.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:30.063+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:30.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:30.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:30.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:30.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:30.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:30.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:30.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:30.102+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:30.380+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:30.380+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:30.380+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:30.380+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:31.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:31.119+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:31.348 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:31.119+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:31.348 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:31.119+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:31.348 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:31.119+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:31.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:31.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:31.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:31.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:31.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:31.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:31.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:31.078+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:31.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:18:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:18:31.501+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 31971 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:18:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:31.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:31.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:31.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:31.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:31.346+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:32.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:32.310+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:32.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:32.310+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:32.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:32.310+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:32.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:32.311+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:32.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:32.168+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:32.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:32.168+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:32.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:32.168+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:32.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:32.169+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:32.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:32.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:32.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:32.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:32.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:32.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:32.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:32.120+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:33.306 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:33.164+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:33.307 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:33.164+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:33.307 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:33.164+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:33.307 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:33.164+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:33.307 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:33.125+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:33.307 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:33.125+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:33.307 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:33.125+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:33.307 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:33.126+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:33.642 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:33.304+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:33.642 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:33.304+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:33.642 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:33.304+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:33.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:33.305+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:34.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:34.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:34.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:34.336+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:34.198+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:34.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:34.198+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:34.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:34.198+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:34.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:34.199+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:34.128+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:34.128+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:34.128+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:34.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:34.129+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:35.327+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:35.327+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:35.327+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:35.328+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:35.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:35.222+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:35.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:35.222+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:35.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:35.222+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:35.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:35.223+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:35.121+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:35.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:35.121+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:35.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:35.121+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:35.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:35.122+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:36.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:36.336+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:36.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:36.336+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:36.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:36.336+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:36.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:36.336+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:36.182+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:36.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:36.182+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:36.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:36.182+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:36.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:36.183+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:36.112+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:36.112+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:36.112+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:36.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:36.113+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:36.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:18:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:18:36.505+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 32050 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:18:37.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:37.317+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:37.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:37.317+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:37.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:37.317+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:37.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:37.317+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:37.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:37.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:37.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:37.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:37.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:37.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:37.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:37.183+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:37.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:37.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:37.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:37.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:37.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:37.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:37.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:37.116+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:38.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:38.179+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:38.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:38.179+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:38.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:38.179+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:38.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:38.180+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:38.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:38.353+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:38.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:38.353+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:38.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:38.353+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:38.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:38.354+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:38.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:38.157+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:38.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:38.157+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:38.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:38.157+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:38.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:38.158+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:39.202+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:39.202+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:39.202+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:39.202+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:39.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:39.338+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:39.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:39.338+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:39.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:39.338+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:39.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:39.339+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:39.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:39.194+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:39.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:39.194+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:39.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:39.194+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:39.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:39.194+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:40.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:40.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:40.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:40.311+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:40.177+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:40.177+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:40.177+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:40.177+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:40.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:40.217+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:40.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:40.217+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:40.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:40.217+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:40.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:40.218+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:41.307+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:41.307+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:41.307+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:41.307+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:41.220+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:41.220+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:41.220+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:41.221+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:41.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:41.209+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:41.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:41.209+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:41.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:41.209+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:41.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:41.210+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:41.856 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:18:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:18:41.509+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 32131 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:18:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:42.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:42.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:42.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:42.311+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:42.233+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:42.233+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:42.233+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:42.233+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:42.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:42.220+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:42.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:42.220+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:42.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:42.220+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:42.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:42.221+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:43.283+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:43.283+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:43.283+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:43.284+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:43.208+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:43.208+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:43.208+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:43.209+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:43.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:43.203+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:43.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:43.203+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:43.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:43.203+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:43.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:43.204+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:44.197+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:44.197+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:44.197+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:44.198+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:44.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:44.289+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:44.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:44.289+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:44.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:44.289+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:44.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:44.289+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:44.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:44.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:44.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:44.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:44.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:44.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:44.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:44.178+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:45.228+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:45.228+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:45.228+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:45.229+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:45.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:45.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:45.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:45.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:45.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:45.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:45.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:45.275+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:45.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:45.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:45.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:45.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:45.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:45.171+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:46.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:46.212+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:46.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:46.212+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:46.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:46.212+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:46.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:46.212+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:46.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:46.281+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:46.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:46.281+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:46.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:46.281+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:46.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:46.281+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:46.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:46.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:46.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:46.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:46.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:46.162+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:46.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:46.163+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:46.862 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:18:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:18:46.513+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 32215 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:18:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:47.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:47.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:47.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:47.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:47.202+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:47.320+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:47.320+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:47.320+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:47.320+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:47.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:47.203+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:47.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:47.204+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:47.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:47.204+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:47.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:47.204+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:48.230+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:48.230+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:48.230+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:48.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:48.231+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:48.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:48.299+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:48.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:48.299+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:48.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:48.299+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:48.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:48.300+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:48.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:48.254+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:48.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:48.254+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:48.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:48.254+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:48.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:48.255+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:49.303+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:49.303+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:49.303+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:49.303+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:49.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:49.309+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:49.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:49.309+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:49.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:49.309+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:49.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:49.309+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:49.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:49.278+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:49.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:49.278+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:49.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:49.278+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:49.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:49.279+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:50.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:50.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:50.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:50.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:50.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:50.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:50.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:50.316+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:50.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:50.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:50.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:50.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:50.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:50.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:50.642 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:50.283+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:50.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:50.344+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:50.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:50.344+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:50.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:50.344+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:50.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:50.345+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:51.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:51.326+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:51.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:51.326+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:51.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:51.326+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:51.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:51.327+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:51.543 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:51.369+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:51.543 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:51.369+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:51.543 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:51.369+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:51.543 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:51.370+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:51.543 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:51.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:51.544 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:51.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:51.544 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:51.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:51.544 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:51.237+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:51.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:18:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:18:51.516+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 32295 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:18:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:52.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:52.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:52.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:52.245+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:52.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:52.363+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:52.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:52.363+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:52.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:52.363+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:52.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:52.364+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:52.401+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:52.401+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:52.401+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:52.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:52.402+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:53.388+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:53.388+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:53.388+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:53.388+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:53.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:53.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:53.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:53.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:53.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:53.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:53.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:53.202+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:53.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:53.397+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:53.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:53.397+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:53.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:53.397+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:53.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:53.397+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:54.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:54.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:54.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:54.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:54.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:54.171+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:54.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:54.172+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:54.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:54.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:54.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:54.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:54.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:54.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:54.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:54.341+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:54.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:54.360+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:54.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:54.360+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:54.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:54.360+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:54.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:54.360+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:55.340+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:55.340+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:55.340+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:55.341+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:55.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:55.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:55.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:55.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:55.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:55.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:55.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:55.188+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:55.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:55.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:55.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:55.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:55.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:55.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:55.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:55.349+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:56.320+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:56.320+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:56.320+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:56.321+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:56.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:56.363+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:56.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:56.363+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:56.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:56.363+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:56.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:56.364+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:56.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:56.195+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:56.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:56.195+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:56.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:56.195+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:56.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:56.195+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:56.878 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:18:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:18:56.522+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 32379 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:18:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:57.339+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:57.339+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:57.339+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:57.340+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:57.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:57.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:57.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:57.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:57.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:57.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:57.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:57.352+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:57.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:57.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:57.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:57.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:57.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:57.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:57.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:57.179+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:58.350+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:58.350+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:58.350+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:58.351+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:58.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:58.196+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:58.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:58.196+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:58.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:58.196+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:58.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:58.197+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:58.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:58.383+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:58.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:58.383+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:58.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:58.383+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:58.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:58.383+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:59.435 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:59.350+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:59.436 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:59.350+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:59.436 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:59.350+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:18:59.436 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:18:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:18:59.351+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:18:59.436 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:59.179+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:59.436 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:59.179+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:59.436 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:59.179+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:18:59.436 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:18:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:18:59.180+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:18:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:59.433+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:59.433+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:59.433+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:18:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:18:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:18:59.433+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:00.140+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:00.140+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:00.140+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:00.140+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:00.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:00.350+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:00.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:00.350+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:00.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:00.350+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:00.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:00.350+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:00.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:00.477+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:00.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:00.477+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:00.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:00.477+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:00.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:00.478+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:01.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:01.123+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:01.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:01.123+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:01.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:01.123+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:01.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:01.124+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:01.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:01.354+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:01.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:01.354+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:01.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:01.354+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:01.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:01.355+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:01.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:19:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:19:01.526+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 32470 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:19:01.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:01.435+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:01.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:01.435+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:01.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:01.435+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:01.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:01.435+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:02.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:02.396+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:02.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:02.396+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:02.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:02.396+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:02.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:02.397+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:02.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:02.150+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:02.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:02.150+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:02.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:02.150+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:02.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:02.151+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:02.928 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:02.480+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:02.929 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:02.480+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:02.929 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:02.480+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:02.929 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:02.481+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:03.462 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:03.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:03.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:03.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:03.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:03.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:03.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:03.413+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:03.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:03.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:03.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:03.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:03.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:03.201+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:03.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:03.202+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:03.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:03.460+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:03.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:03.460+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:03.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:03.460+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:03.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:03.461+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:04.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:04.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:04.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:04.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:04.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:04.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:04.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:04.409+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:04.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:04.167+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:04.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:04.167+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:04.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:04.167+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:04.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:04.168+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:04.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:04.446+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:04.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:04.446+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:04.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:04.446+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:04.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:04.447+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:05.411 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:05.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:05.411 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:05.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:05.411 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:05.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:05.411 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:05.139+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:05.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:05.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:05.408+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:05.409+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:05.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:05.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:05.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:05.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:05.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:05.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:05.478+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:06.429 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:06.173+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:06.429 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:06.173+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:06.429 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:06.173+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:06.429 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:06.174+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:06.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:19:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:19:06.530+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 32558 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:19:06.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:06.505+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:06.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:06.505+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:06.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:06.505+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:06.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:06.505+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:06.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:06.426+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:06.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:06.426+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:06.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:06.426+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:06.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:06.427+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:07.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:07.419+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:07.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:07.419+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:07.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:07.419+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:07.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:07.420+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:07.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:07.218+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:07.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:07.219+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:07.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:07.219+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:07.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:07.219+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:07.522+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:07.522+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:07.522+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:07.523+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:08.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:08.431+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:08.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:08.431+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:08.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:08.431+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:08.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:08.432+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:08.189+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:08.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:08.189+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:08.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:08.189+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:08.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:08.190+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:08.528+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:08.528+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:08.528+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:08.528+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:09.441 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:09.152+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:09.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:09.152+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:09.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:09.152+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:09.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:09.153+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:09.439+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:09.439+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:09.439+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:09.440+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:09.485+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:09.485+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:09.485+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:09.485+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:10.411 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:10.124+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:10.411 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:10.124+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:10.412 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:10.124+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:10.412 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:10.125+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:10.409+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:10.409+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:10.409+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:10.410+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:10.487+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:10.487+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:10.487+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:10.488+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:11.426 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:11.164+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:11.426 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:11.164+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:11.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:11.164+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:11.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:11.165+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:11.719 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:19:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:19:11.534+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 32650 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:19:11.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:11.424+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:11.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:11.424+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:11.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:11.424+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:11.720 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:11.424+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:11.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:11.463+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:11.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:11.463+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:11.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:11.463+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:11.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:11.464+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:12.443 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:12.200+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:12.443 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:12.200+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:12.443 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:12.200+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:12.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:12.200+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:12.441+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:12.441+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:12.441+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:12.442+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:12.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:12.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:12.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:12.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:12.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:12.441+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:13.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:13.426+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:13.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:13.426+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:13.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:13.426+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:13.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:13.426+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:13.645 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:13.470+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:13.645 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:13.470+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:13.645 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:13.470+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:13.646 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:13.470+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:13.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:13.235+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:13.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:13.235+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:13.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:13.235+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:13.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:13.236+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:14.220+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:14.220+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:14.220+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:14.221+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:14.409+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:14.409+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:14.409+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:14.410+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:14.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:14.439+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:14.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:14.439+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:14.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:14.439+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:14.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:14.439+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:15.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:15.206+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:15.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:15.206+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:15.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:15.206+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:15.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:15.206+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:15.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:15.444+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:15.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:15.444+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:15.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:15.444+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:15.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:15.445+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:15.460+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:15.460+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:15.460+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:15.461+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:16.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:16.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:16.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:16.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:16.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:16.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:16.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:16.435+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:16.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:16.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:16.543 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:16.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:16.543 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:16.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:16.543 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:16.229+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:16.543 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:16.415+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:16.543 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:16.415+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:16.543 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:16.415+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:16.543 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:16.415+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:16.875 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:19:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:19:16.539+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 32738 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:19:17.471 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:17.232+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:17.471 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:17.232+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:17.471 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:17.232+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:17.471 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:17.232+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:17.472 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:17.403+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:17.472 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:17.403+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:17.472 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:17.403+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:17.472 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:17.404+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:17.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:17.468+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:17.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:17.468+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:17.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:17.468+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:17.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:17.469+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:18.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:18.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:18.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:18.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:18.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:18.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:18.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:18.200+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:18.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:18.416+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:18.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:18.416+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:18.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:18.416+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:18.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:18.416+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:18.842 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:18.507+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:18.842 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:18.507+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:18.842 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:18.507+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:18.842 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:18.508+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:19.498 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:19.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:19.499 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:19.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:19.499 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:19.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:19.499 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:19.238+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:19.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:19.406+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:19.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:19.406+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:19.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:19.406+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:19.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:19.407+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:19.496+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:19.496+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:19.496+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:19.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:19.497+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:20.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:20.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:20.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:20.193+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:20.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:20.400+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:20.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:20.400+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:20.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:20.400+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:20.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:20.401+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:20.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:20.541+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:20.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:20.541+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:20.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:20.541+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:20.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:20.541+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:21.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:21.222+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:21.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:21.222+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:21.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:21.222+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:21.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:21.222+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:21.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:21.413+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:21.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:21.413+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:21.540 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:21.413+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:21.540 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:21.414+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:21.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:19:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:19:21.543+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 32829 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:19:21.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:21.536+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:21.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:21.536+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:21.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:21.536+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:21.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:21.537+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:22.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:22.266+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:22.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:22.266+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:22.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:22.266+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:22.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:22.266+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:22.540 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:22.403+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:22.541 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:22.403+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:22.541 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:22.403+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:22.541 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:22.403+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:22.535+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:22.535+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:22.535+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:22.536+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:23.585 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:23.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:23.585 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:23.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:23.585 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:23.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:23.586 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:23.238+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:23.586 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:23.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:23.586 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:23.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:23.586 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:23.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:23.586 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:23.385+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:23.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:23.581+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:23.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:23.581+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:23.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:23.581+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:23.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:23.582+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:24.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:24.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:24.611 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:24.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:24.611 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:24.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:24.611 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:24.272+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:24.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:24.414+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:24.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:24.414+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:24.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:24.414+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:24.612 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:24.415+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:24.608+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:24.608+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:24.608+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:24.609+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:25.639 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:25.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:25.639 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:25.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:25.639 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:25.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:25.639 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:25.372+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:25.640 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:25.244+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:25.640 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:25.244+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:25.640 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:25.244+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:25.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:25.244+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:25.636+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:25.636+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:25.636+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:25.637+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:26.550 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:26.231+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:26.551 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:26.231+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:26.551 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:26.231+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:26.551 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:26.232+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:26.551 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:26.408+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:26.551 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:26.408+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:26.551 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:26.408+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:26.552 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:26.409+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:26.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:19:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:19:26.547+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 32917 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:19:26.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:26.618+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:26.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:26.618+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:26.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:26.618+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:26.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:26.619+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:27.185+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:27.185+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:27.185+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:27.186+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:27.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:27.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:27.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:27.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:27.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:27.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:27.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:27.410+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:27.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:27.569+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:27.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:27.569+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:27.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:27.569+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:27.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:27.570+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:28.584 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:28.231+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:28.585 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:28.231+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:28.585 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:28.231+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:28.585 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:28.232+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:28.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:28.435+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:28.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:28.435+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:28.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:28.435+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:28.586 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:28.436+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:28.582+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:28.582+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:28.582+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:28.582+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:29.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:29.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:29.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:29.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:29.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:29.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:29.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:29.273+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:29.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:29.459+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:29.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:29.459+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:29.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:29.459+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:29.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:29.460+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:29.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:29.607+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:29.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:29.607+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:29.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:29.607+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:29.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:29.608+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:30.496 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:30.259+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:30.496 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:30.259+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:30.497 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:30.259+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:30.497 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:30.260+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:30.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:30.581+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:30.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:30.581+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:30.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:30.581+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:30.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:30.582+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:30.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:30.494+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:30.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:30.494+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:30.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:30.494+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:30.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:30.495+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:31.555 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:31.215+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:31.555 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:31.215+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:31.555 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:31.215+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:31.555 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:31.215+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:31.556 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:31.452+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:31.556 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:31.452+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:31.556 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:31.452+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:31.556 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:31.452+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:31.858 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:19:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:19:31.552+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 33009 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:19:31.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:31.604+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:31.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:31.604+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:31.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:31.604+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:31.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:31.605+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:32.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:32.230+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:32.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:32.230+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:32.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:32.230+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:32.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:32.230+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:32.626 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:32.458+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:32.626 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:32.458+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:32.626 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:32.458+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:32.626 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:32.458+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:32.622+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:32.622+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:32.622+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:32.623+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:33.629 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:33.278+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:33.629 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:33.278+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:33.629 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:33.278+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:33.629 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:33.279+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:33.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:33.412+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:33.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:33.412+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:33.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:33.412+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:33.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:33.413+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:33.626+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:33.626+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:33.626+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:33.627+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:34.398 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:34.250+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:34.398 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:34.250+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:34.398 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:34.250+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:34.398 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:34.251+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:34.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:34.395+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:34.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:34.395+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:34.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:34.395+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:34.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:34.396+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:34.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:34.669+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:34.669+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:34.669+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:35.652 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:35.431+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:35.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:35.431+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:35.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:35.431+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:35.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:35.431+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:35.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:35.225+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:35.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:35.225+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:35.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:35.225+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:35.654 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:35.226+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:35.649+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:35.649+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:35.649+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:35.649+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:36.559 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:36.239+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:36.559 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:36.239+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:36.559 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:36.239+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:36.559 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:36.240+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:36.559 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:36.447+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:36.559 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:36.447+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:36.560 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:36.447+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:36.560 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:36.448+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:36.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:19:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:19:36.555+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 33097 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:19:36.872 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:36.676+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:36.872 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:36.676+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:36.872 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:36.676+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:36.872 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:36.677+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:37.478 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:37.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:37.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:37.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:37.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:37.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:37.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:37.221+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:37.475+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:37.475+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:37.475+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:37.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:37.475+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:37.701+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:37.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:37.701+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:37.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:37.701+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:37.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:37.702+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:38.509 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:38.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:38.509 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:38.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:38.509 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:38.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:38.510 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:38.252+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:38.685+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:38.685+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:38.685+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:38.686+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:38.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:38.506+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:38.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:38.506+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:38.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:38.506+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:38.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:38.506+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:39.226+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:39.226+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:39.226+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:39.227+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:39.556+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:39.556+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:39.556+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:39.556+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:39.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:39.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:39.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:39.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:39.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:39.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:39.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:39.667+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:40.179+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:40.179+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:40.179+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:40.180+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:40.566+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:40.566+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:40.566+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:40.566+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:40.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:40.656+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:40.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:40.656+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:40.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:40.656+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:40.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:40.656+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:41.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:41.194+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:41.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:41.194+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:41.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:41.194+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:41.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:41.195+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:41.858 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:19:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:19:41.560+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 33188 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:19:41.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:41.631+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:41.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:41.631+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:41.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:41.631+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:41.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:41.632+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:41.859 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:41.566+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:41.859 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:41.566+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:41.859 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:41.566+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:41.859 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:41.568+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:42.529 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:42.227+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:42.529 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:42.227+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:42.529 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:42.227+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:42.529 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:42.228+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:42.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:42.526+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:42.865 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:42.526+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:42.865 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:42.526+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:42.865 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:42.527+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:42.871 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:42.673+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:42.871 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:42.673+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:42.872 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:42.673+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:42.872 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:42.674+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:43.546 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:43.241+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:43.546 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:43.241+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:43.547 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:43.241+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:43.547 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:43.242+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:43.544+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:43.544+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:43.544+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:43.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:43.544+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:43.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:43.626+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:43.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:43.626+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:43.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:43.626+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:43.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:43.627+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:44.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:44.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:44.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:44.204+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:44.619+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:44.619+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:44.619+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:44.620+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:44.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:44.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:44.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:44.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:44.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:44.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:44.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:44.575+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:45.186+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:45.186+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:45.186+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:45.187+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:45.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:45.591+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:45.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:45.591+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:45.937 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:45.591+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:45.937 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:45.592+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:45.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:45.546+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:45.938 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:45.546+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:45.938 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:45.546+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:45.938 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:45.547+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:46.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:46.152+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:46.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:46.152+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:46.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:46.152+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:46.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:46.153+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:46.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:19:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:19:46.564+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 33275 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:19:46.870 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:46.636+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:46.871 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:46.636+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:46.871 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:46.636+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:46.871 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:46.637+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:46.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:46.553+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:46.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:46.553+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:46.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:46.553+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:46.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:46.554+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:47.140+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:47.140+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:47.140+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:47.140+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:47.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:47.616+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:47.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:47.616+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:47.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:47.616+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:47.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:47.617+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:47.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:47.516+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:47.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:47.516+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:47.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:47.516+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:47.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:47.517+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:48.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:48.119+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:48.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:48.119+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:48.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:48.119+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:48.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:48.119+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:48.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:48.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:48.667+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:48.667+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:48.474+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:48.474+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:48.474+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:48.475+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:49.321 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:49.127+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:49.321 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:49.127+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:49.321 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:49.127+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:49.322 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:49.128+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:49.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:49.473+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:49.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:49.473+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:49.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:49.473+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:49.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:49.474+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:49.699+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:49.699+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:49.699+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:49.700+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:50.160+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:50.160+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:50.160+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:50.161+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:50.753 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:50.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:50.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:50.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:50.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:50.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:50.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:50.669+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:50.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:50.499+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:50.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:50.499+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:50.499+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:50.499+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:51.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:51.173+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:51.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:51.173+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:51.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:51.173+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:51.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:51.174+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:51.762 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:19:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:19:51.568+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 33367 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:19:51.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:51.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:51.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:51.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:51.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:51.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:51.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:51.719+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:51.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:51.500+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:51.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:51.500+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:51.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:51.500+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:51.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:51.501+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:52.219+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:52.219+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:52.219+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:52.219+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:52.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:52.493+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:52.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:52.493+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:52.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:52.493+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:52.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:52.494+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:52.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:52.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:52.764+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:52.765+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:53.543 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:53.247+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:53.544 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:53.247+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:53.544 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:53.247+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:53.544 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:53.248+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:53.541+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:53.541+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:53.541+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:53.542+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:53.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:53.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:53.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:53.782+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:53.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:53.783+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:54.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:54.252+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:54.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:54.252+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:54.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:54.252+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:54.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:54.253+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:54.789 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:54.515+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:54.789 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:54.515+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:54.789 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:54.515+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:54.789 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:54.516+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:55.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:54.786+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:55.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:54.786+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:55.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:54.786+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:55.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:54.787+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:55.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:55.262+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:55.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:55.262+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:55.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:55.262+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:55.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:55.263+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:55.837 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:55.483+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:55.838 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:55.483+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:55.838 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:55.483+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:55.838 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:55.484+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:55.835+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:55.835+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:55.835+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:55.835+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:56.576 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:56.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:56.576 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:56.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:56.576 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:56.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:56.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:56.222+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:56.577 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:56.464+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:56.577 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:56.464+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:56.577 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:56.464+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:56.577 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:56.464+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:56.855 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:19:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:19:56.573+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 33455 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:19:57.200 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:56.852+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:57.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:56.852+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:57.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:56.852+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:57.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:56.853+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:57.198+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:57.198+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:57.198+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:57.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:57.199+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:57.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:57.484+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:57.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:57.484+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:57.875 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:57.484+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:57.875 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:57.485+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:58.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:57.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:58.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:57.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:58.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:57.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:58.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:57.873+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:58.152+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:58.152+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:58.152+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:58.153+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:58.866 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:58.496+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:58.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:58.496+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:58.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:58.496+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:58.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:58.496+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:59.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:58.864+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:59.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:58.864+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:59.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:58.864+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:19:59.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:58.865+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:19:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:59.168+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:59.168+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:59.168+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:19:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:19:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:19:59.169+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:19:59.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:59.490+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:59.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:59.490+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:59.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:59.490+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:19:59.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:19:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:19:59.491+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:00.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:59.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:00.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:59.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:00.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:59.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:00.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:19:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:19:59.859+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:00.209+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:00.209+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:00.209+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:00.209+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:00.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:00.448+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:00.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:00.448+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:00.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:00.448+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:00.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:00.449+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:01.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:00.843+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:01.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:00.843+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:01.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:00.843+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:01.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:00.843+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:01.207+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:01.207+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:01.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:01.207+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:01.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:01.207+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:01.411+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:01.411+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:01.411+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:01.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:01.411+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:01.854 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:20:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:20:01.577+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 33546 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:20:02.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:01.852+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:02.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:01.852+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:02.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:01.852+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:02.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:01.853+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:02.455 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:02.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:02.455 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:02.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:02.455 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:02.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:02.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:02.176+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:02.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:02.452+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:02.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:02.452+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:02.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:02.452+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:02.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:02.453+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:03.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:02.837+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:03.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:02.837+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:03.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:02.837+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:03.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:02.837+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:03.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:03.150+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:03.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:03.150+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:03.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:03.150+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:03.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:03.151+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:03.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:03.449+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:03.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:03.449+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:03.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:03.449+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:03.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:03.450+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:04.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:03.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:04.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:03.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:04.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:03.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:04.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:03.881+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:04.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:04.163+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:04.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:04.163+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:04.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:04.163+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:04.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:04.164+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:04.846 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:04.493+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:04.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:04.493+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:04.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:04.493+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:04.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:04.493+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:05.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:04.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:05.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:04.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:05.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:04.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:05.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:04.844+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:05.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:05.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:05.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:05.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:05.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:05.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:05.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:05.174+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:05.841 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:05.471+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:05.842 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:05.471+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:05.842 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:05.471+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:05.842 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:05.472+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:06.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:05.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:06.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:05.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:06.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:05.839+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:06.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:05.840+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:06.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:06.205+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:06.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:06.205+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:06.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:06.205+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:06.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:06.205+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:06.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:06.428+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:06.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:06.428+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:06.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:06.428+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:06.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:06.429+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:06.846 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:20:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:20:06.582+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 33634 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:20:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:06.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:06.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:06.844+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:06.845+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:07.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:07.399+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:07.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:07.399+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:07.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:07.399+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:07.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:07.399+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:07.241+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:07.241+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:07.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:07.241+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:07.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:07.242+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:07.825+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:07.825+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:07.825+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:07.825+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:08.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:08.411+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:08.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:08.411+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:08.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:08.411+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:08.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:08.412+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:08.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:08.233+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:08.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:08.233+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:08.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:08.233+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:08.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:08.233+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:09.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:08.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:09.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:08.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:09.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:08.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:09.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:08.856+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:09.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:09.191+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:09.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:09.191+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:09.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:09.191+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:09.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:09.192+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:09.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:09.396+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:09.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:09.396+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:09.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:09.396+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:09.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:09.396+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:09.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:09.845+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:09.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:09.845+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:09.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:09.845+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:09.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:09.846+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:10.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:10.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:10.445 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:10.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:10.445 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:10.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:10.445 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:10.158+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:10.442+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:10.442+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:10.442+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:10.442+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:11.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:10.825+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:11.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:10.825+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:11.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:10.825+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:11.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:10.825+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:11.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:11.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:11.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:11.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:11.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:11.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:11.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:11.140+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:11.762 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:20:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:20:11.586+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 33726 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:20:11.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:11.491+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:11.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:11.491+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:11.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:11.491+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:11.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:11.491+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:12.105 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:11.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:12.105 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:11.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:12.106 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:11.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:12.106 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:11.841+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:12.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:12.103+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:12.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:12.103+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:12.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:12.103+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:12.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:12.103+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:12.460+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:12.460+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:12.460+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:12.461+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:12.843+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:12.843+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:12.843+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:12.844+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:13.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:13.055+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:13.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:13.055+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:13.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:13.055+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:13.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:13.056+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:13.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:13.464+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:13.877 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:13.464+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:13.877 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:13.464+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:13.877 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:13.465+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:14.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:14.015+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:14.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:14.015+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:14.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:14.015+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:14.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:14.015+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:13.874+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:14.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:13.874+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:14.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:13.874+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:14.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:13.874+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:14.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:14.496+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:14.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:14.496+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:14.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:14.496+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:14.928 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:14.497+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:14.924+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:14.924+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:14.924+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:14.925+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:15.052+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:15.052+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:15.052+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:15.053+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:15.453+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:15.453+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:15.453+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:15.454+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:16.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:16.069+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:16.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:16.069+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:16.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:16.069+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:16.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:16.070+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:16.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:15.931+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:16.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:15.931+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:16.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:15.931+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:16.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:15.932+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:16.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:20:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:20:16.590+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 33815 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:20:16.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:16.462+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:16.462+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:16.462+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:16.463+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:17.078+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:17.078+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:17.078+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:17.079+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:16.941+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:16.941+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:17.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:16.941+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:17.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:16.942+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:17.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:17.429+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:17.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:17.429+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:17.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:17.429+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:17.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:17.430+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:17.950+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:17.950+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:17.950+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:17.951+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:18.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:18.097+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:18.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:18.097+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:18.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:18.097+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:18.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:18.098+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:18.398+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:18.398+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:18.398+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:18.399+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:19.142+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:19.142+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:19.142+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:19.142+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:18.911+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:19.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:18.911+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:19.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:18.911+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:19.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:18.911+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:19.357+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:19.357+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:19.357+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:19.358+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:20.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:19.937+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:20.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:19.937+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:20.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:19.937+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:20.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:19.938+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:20.397 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:20.100+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:20.397 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:20.100+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:20.397 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:20.100+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:20.397 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:20.101+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:20.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:20.394+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:20.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:20.394+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:20.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:20.394+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:20.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:20.395+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:21.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:20.964+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:21.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:20.964+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:21.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:20.964+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:21.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:20.965+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:21.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:21.145+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:21.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:21.145+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:21.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:21.145+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:21.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:21.145+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:21.597 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:21.372+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:21.598 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:21.372+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:21.598 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:21.372+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:21.598 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:21.373+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:21.862 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:20:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:20:21.594+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 33906 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:20:22.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:22.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:22.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:22.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:22.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:22.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:22.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:22.001+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:22.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:22.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:22.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:22.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:22.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:22.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:22.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:22.119+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:22.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:22.383+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:22.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:22.383+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:22.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:22.383+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:22.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:22.384+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:23.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:22.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:23.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:22.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:23.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:22.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:23.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:22.963+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:23.144+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:23.144+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:23.144+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:23.144+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:23.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:23.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:23.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:23.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:23.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:23.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:23.360+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:24.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:23.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:24.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:23.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:24.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:23.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:24.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:23.931+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:24.360+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:24.360+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:24.360+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:24.360+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:24.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:24.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:24.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:24.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:24.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:24.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:24.184+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:25.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:24.934+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:25.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:24.934+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:25.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:24.934+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:25.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:24.935+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:25.165+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:25.165+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:25.165+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:25.165+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:25.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:25.356+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:25.356+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:25.356+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:25.356+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:26.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:25.916+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:26.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:25.916+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:26.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:25.916+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:26.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:25.917+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:26.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:26.125+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:26.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:26.125+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:26.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:26.125+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:26.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:26.125+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:26.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:26.354+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:26.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:26.354+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:26.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:26.354+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:26.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:26.354+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:26.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:20:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:20:26.599+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 33995 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:20:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:26.911+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:26.911+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:26.911+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:26.911+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:27.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:27.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:27.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:27.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:27.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:27.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:27.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:27.118+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:27.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-20T20:20:27.219+0000 7fb660996640 0 monclient(hunting): authenticate timed out after 300 2024-09-20T20:20:27.223 INFO:teuthology.orchestra.run.smithi114.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-09-20T20:20:27.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:27.399+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:27.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:27.399+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:27.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:27.399+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:27.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:27.400+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T20:20:28.002 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-09-20T20:20:28.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T20:20:28.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-09-20T20:20:28.059 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/open/open10.c: Cannot open: Input/output error 2024-09-20T20:20:28.059 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/open/open06.c: Cannot open: Input/output error 2024-09-20T20:20:28.059 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/open/open04.c: Cannot open: Input/output error 2024-09-20T20:20:28.059 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/open/open05.c: Cannot open: Input/output error 2024-09-20T20:20:28.059 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/open/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.059 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/open/open09.c: Cannot open: Input/output error 2024-09-20T20:20:28.060 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/open/open01.c: Cannot open: Input/output error 2024-09-20T20:20:28.060 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/open/open02.c: Cannot open: Input/output error 2024-09-20T20:20:28.060 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/open/open08.c: Cannot open: Input/output error 2024-09-20T20:20:28.060 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/open/open07.c: Cannot open: Input/output error 2024-09-20T20:20:28.060 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/open/open03.c: Cannot open: Input/output error 2024-09-20T20:20:28.060 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/open: Cannot utime: Input/output error 2024-09-20T20:20:28.060 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/syslog: Cannot mkdir: Input/output error 2024-09-20T20:20:28.060 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/syslog/syslog05: Cannot open: Input/output error 2024-09-20T20:20:28.060 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/syslog/syslog02: Cannot open: Input/output error 2024-09-20T20:20:28.061 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/syslog/syslogtst.c: Cannot open: Input/output error 2024-09-20T20:20:28.061 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/syslog/syslog06: Cannot open: Input/output error 2024-09-20T20:20:28.061 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/syslog/syslog11.c: Cannot open: Input/output error 2024-09-20T20:20:28.061 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/syslog/syslog10: Cannot open: Input/output error 2024-09-20T20:20:28.061 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/syslog/syslog01: Cannot open: Input/output error 2024-09-20T20:20:28.061 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/syslog/syslog04: Cannot open: Input/output error 2024-09-20T20:20:28.061 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/syslog/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.062 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/syslog/syslog07: Cannot open: Input/output error 2024-09-20T20:20:28.062 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/syslog/syslog03: Cannot open: Input/output error 2024-09-20T20:20:28.062 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/syslog/syslog09: Cannot open: Input/output error 2024-09-20T20:20:28.062 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/syslog/syslog08: Cannot open: Input/output error 2024-09-20T20:20:28.062 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/syslog/syslog12.c: Cannot open: Input/output error 2024-09-20T20:20:28.062 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getcontext: Cannot mkdir: Input/output error 2024-09-20T20:20:28.063 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getcontext/getcontext01.c: Cannot open: Input/output error 2024-09-20T20:20:28.063 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getcontext/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.063 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/switch: Cannot mkdir: Input/output error 2024-09-20T20:20:28.063 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/switch/endian_switch01.c: Cannot open: Input/output error 2024-09-20T20:20:28.063 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/switch/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.063 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chmod: Cannot mkdir: Input/output error 2024-09-20T20:20:28.063 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chmod/chmod06.c: Cannot open: Input/output error 2024-09-20T20:20:28.064 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chmod/chmod01.c: Cannot open: Input/output error 2024-09-20T20:20:28.064 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chmod/chmod07.c: Cannot open: Input/output error 2024-09-20T20:20:28.064 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chmod/chmod03.c: Cannot open: Input/output error 2024-09-20T20:20:28.064 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chmod/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.064 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chmod/chmod02.c: Cannot open: Input/output error 2024-09-20T20:20:28.064 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chmod/change_owner.mode.sh: Cannot open: Input/output error 2024-09-20T20:20:28.064 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chmod/change_owner.c: Cannot open: Input/output error 2024-09-20T20:20:28.065 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chmod/chmod04.c: Cannot open: Input/output error 2024-09-20T20:20:28.065 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/chmod/chmod05.c: Cannot open: Input/output error 2024-09-20T20:20:28.065 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/capget: Cannot mkdir: Input/output error 2024-09-20T20:20:28.065 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/capget/capget01.c: Cannot open: Input/output error 2024-09-20T20:20:28.065 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/capget/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.065 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/capget/capget02.c: Cannot open: Input/output error 2024-09-20T20:20:28.065 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll: Cannot mkdir: Input/output error 2024-09-20T20:20:28.065 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll/README.1ST: Cannot open: Input/output error 2024-09-20T20:20:28.066 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.066 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll/epoll-ltp.c: Cannot open: Input/output error 2024-09-20T20:20:28.066 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysctl: Cannot mkdir: Input/output error 2024-09-20T20:20:28.066 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysctl/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.066 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysctl/sysctl03.c: Cannot open: Input/output error 2024-09-20T20:20:28.066 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysctl/sysctl04.c: Cannot open: Input/output error 2024-09-20T20:20:28.066 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysctl/sysctl05.c: Cannot open: Input/output error 2024-09-20T20:20:28.066 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysctl/sysctl01.c: Cannot open: Input/output error 2024-09-20T20:20:28.067 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getpriority: Cannot mkdir: Input/output error 2024-09-20T20:20:28.067 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getpriority/getpriority02.c: Cannot open: Input/output error 2024-09-20T20:20:28.067 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getpriority/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.067 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getpriority/getpriority01.c: Cannot open: Input/output error 2024-09-20T20:20:28.067 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/munlockall: Cannot mkdir: Input/output error 2024-09-20T20:20:28.067 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/munlockall/munlockall02.c: Cannot open: Input/output error 2024-09-20T20:20:28.067 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/munlockall/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.067 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/munlockall/munlockall01.c: Cannot open: Input/output error 2024-09-20T20:20:28.067 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getsid: Cannot mkdir: Input/output error 2024-09-20T20:20:28.068 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getsid/getsid01.c: Cannot open: Input/output error 2024-09-20T20:20:28.068 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getsid/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.068 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getsid/getsid02.c: Cannot open: Input/output error 2024-09-20T20:20:28.068 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchdir: Cannot mkdir: Input/output error 2024-09-20T20:20:28.068 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchdir/fchdir02.c: Cannot open: Input/output error 2024-09-20T20:20:28.068 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchdir/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.068 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchdir/fchdir01.c: Cannot open: Input/output error 2024-09-20T20:20:28.068 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchdir/fchdir03.c: Cannot open: Input/output error 2024-09-20T20:20:28.069 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sync: Cannot mkdir: Input/output error 2024-09-20T20:20:28.069 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sync/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.069 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sync/sync02.c: Cannot open: Input/output error 2024-09-20T20:20:28.069 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sync/sync01.c: Cannot open: Input/output error 2024-09-20T20:20:28.069 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/memcmp: Cannot mkdir: Input/output error 2024-09-20T20:20:28.069 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/memcmp/memcmp01.c: Cannot open: Input/output error 2024-09-20T20:20:28.069 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/memcmp/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.070 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vmsplice: Cannot mkdir: Input/output error 2024-09-20T20:20:28.070 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vmsplice/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.070 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vmsplice/vmsplice01.c: Cannot open: Input/output error 2024-09-20T20:20:28.070 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fmtmsg: Cannot mkdir: Input/output error 2024-09-20T20:20:28.070 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fmtmsg/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.070 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fmtmsg/fmtmsg01.c: Cannot open: Input/output error 2024-09-20T20:20:28.071 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2: Cannot mkdir: Input/output error 2024-09-20T20:20:28.071 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/man: Cannot mkdir: Input/output error 2024-09-20T20:20:28.071 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/man/epoll_create.ps: Cannot open: Input/output error 2024-09-20T20:20:28.071 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/man/epoll.4: Cannot open: Input/output error 2024-09-20T20:20:28.071 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/man/epoll.ps: Cannot open: Input/output error 2024-09-20T20:20:28.071 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/man/epoll_create.txt: Cannot open: Input/output error 2024-09-20T20:20:28.071 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/man/epoll_wait.txt: Cannot open: Input/output error 2024-09-20T20:20:28.072 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/man/epoll_wait.2: Cannot open: Input/output error 2024-09-20T20:20:28.072 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/man/epoll.txt: Cannot open: Input/output error 2024-09-20T20:20:28.072 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/man/epoll_create.2: Cannot open: Input/output error 2024-09-20T20:20:28.072 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/man/epoll_wait.ps: Cannot open: Input/output error 2024-09-20T20:20:28.072 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/man/mkpages.sh: Cannot open: Input/output error 2024-09-20T20:20:28.073 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/man/epoll_ctl.2: Cannot open: Input/output error 2024-09-20T20:20:28.073 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/man/epoll_ctl.txt: Cannot open: Input/output error 2024-09-20T20:20:28.074 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-09-20T20:20:28.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-20T20:20:28.075 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-20T20:20:28.081 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-20T20:20:28.082 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/man/epoll_ctl.ps: Cannot open: Input/output error 2024-09-20T20:20:28.082 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/examples: Cannot mkdir: Input/output error 2024-09-20T20:20:28.082 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/examples/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.082 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/examples/epoll-test.c: Cannot open: Input/output error 2024-09-20T20:20:28.082 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/src: Cannot mkdir: Input/output error 2024-09-20T20:20:28.082 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/src/epoll.c: Cannot open: Input/output error 2024-09-20T20:20:28.082 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/include: Cannot mkdir: Input/output error 2024-09-20T20:20:28.083 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/include/epoll.h: Cannot open: Input/output error 2024-09-20T20:20:28.083 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/epoll2/include/dbllist.h: Cannot open: Input/output error 2024-09-20T20:20:28.083 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/quotactl: Cannot mkdir: Input/output error 2024-09-20T20:20:28.083 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/quotactl/quotactl01.c: Cannot open: Input/output error 2024-09-20T20:20:28.083 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/quotactl/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.083 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/truncate: Cannot mkdir: Input/output error 2024-09-20T20:20:28.083 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/truncate/truncate04.c: Cannot open: Input/output error 2024-09-20T20:20:28.084 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/truncate/truncate02.c: Cannot open: Input/output error 2024-09-20T20:20:28.084 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/truncate/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.084 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/truncate/truncate03.c: Cannot open: Input/output error 2024-09-20T20:20:28.084 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/truncate/truncate01.c: Cannot open: Input/output error 2024-09-20T20:20:28.085 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/profil: Cannot mkdir: Input/output error 2024-09-20T20:20:28.085 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/profil/profil01.c: Cannot open: Input/output error 2024-09-20T20:20:28.085 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/profil/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.085 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/timer_getoverrun: Cannot mkdir: Input/output error 2024-09-20T20:20:28.085 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/timer_getoverrun/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.086 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/timer_getoverrun/timer_getoverrun01.c: Cannot open: Input/output error 2024-09-20T20:20:28.086 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/msync: Cannot mkdir: Input/output error 2024-09-20T20:20:28.086 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/msync/msync04.c: Cannot open: Input/output error 2024-09-20T20:20:28.086 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/msync/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.087 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/msync/msync01.c: Cannot open: Input/output error 2024-09-20T20:20:28.087 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/msync/msync02.c: Cannot open: Input/output error 2024-09-20T20:20:28.087 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/msync/msync03.c: Cannot open: Input/output error 2024-09-20T20:20:28.087 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/msync/msync05.c: Cannot open: Input/output error 2024-09-20T20:20:28.088 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigprocmask: Cannot mkdir: Input/output error 2024-09-20T20:20:28.088 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigprocmask/sigprocmask01.c: Cannot open: Input/output error 2024-09-20T20:20:28.088 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigprocmask/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.088 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaltstack: Cannot mkdir: Input/output error 2024-09-20T20:20:28.088 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaltstack/sigaltstack02.c: Cannot open: Input/output error 2024-09-20T20:20:28.089 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaltstack/sigaltstack01.c: Cannot open: Input/output error 2024-09-20T20:20:28.089 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaltstack/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.089 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs: Cannot mkdir: Input/output error 2024-09-20T20:20:28.089 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs/sysfs02.c: Cannot open: Input/output error 2024-09-20T20:20:28.090 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs/sysfs05.c: Cannot open: Input/output error 2024-09-20T20:20:28.090 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.090 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs/sysfs06.c: Cannot open: Input/output error 2024-09-20T20:20:28.090 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs/sysfs03.c: Cannot open: Input/output error 2024-09-20T20:20:28.090 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs/sysfs04.c: Cannot open: Input/output error 2024-09-20T20:20:28.091 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysfs/sysfs01.c: Cannot open: Input/output error 2024-09-20T20:20:28.091 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/readlinkat: Cannot mkdir: Input/output error 2024-09-20T20:20:28.091 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/readlinkat/readlinkat01.c: Cannot open: Input/output error 2024-09-20T20:20:28.091 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/readlinkat/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.091 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/wait: Cannot mkdir: Input/output error 2024-09-20T20:20:28.092 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/wait/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.092 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/wait/wait02.c: Cannot open: Input/output error 2024-09-20T20:20:28.092 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketcall: Cannot mkdir: Input/output error 2024-09-20T20:20:28.092 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketcall/socketcall03.c: Cannot open: Input/output error 2024-09-20T20:20:28.092 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketcall/socketcall04.c: Cannot open: Input/output error 2024-09-20T20:20:28.093 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketcall/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.093 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketcall/socketcall01.c: Cannot open: Input/output error 2024-09-20T20:20:28.093 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/socketcall/socketcall02.c: Cannot open: Input/output error 2024-09-20T20:20:28.093 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchownat: Cannot mkdir: Input/output error 2024-09-20T20:20:28.094 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchownat/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.094 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchownat/fchownat01.c: Cannot open: Input/output error 2024-09-20T20:20:28.094 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup2: Cannot mkdir: Input/output error 2024-09-20T20:20:28.094 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup2/dup203.c: Cannot open: Input/output error 2024-09-20T20:20:28.095 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup2/dup204.c: Cannot open: Input/output error 2024-09-20T20:20:28.095 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup2/dup205.c: Cannot open: Input/output error 2024-09-20T20:20:28.095 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup2/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.095 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup2/dup201.c: Cannot open: Input/output error 2024-09-20T20:20:28.096 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup2/dup202.c: Cannot open: Input/output error 2024-09-20T20:20:28.096 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sbrk: Cannot mkdir: Input/output error 2024-09-20T20:20:28.096 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sbrk/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.096 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sbrk/sbrk01.c: Cannot open: Input/output error 2024-09-20T20:20:28.097 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unshare: Cannot mkdir: Input/output error 2024-09-20T20:20:28.097 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unshare/Makefile: Cannot open: Input/output error 2024-09-20T20:20:28.097 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unshare/unshare01.c: Cannot open: Input/output error 2024-09-20T20:20:28.097 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-20T20:20:28.097 INFO:tasks.workunit.client.0.smithi114.stderr:gzip: stdin: Stale file handle 2024-09-20T20:20:28.098 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unshare/unshare02.c: Cannot open: Input/output error 2024-09-20T20:20:28.098 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fadvise: Cannot mkdir: Input/output error 2024-09-20T20:20:28.098 INFO:tasks.workunit.client.0.smithi114.stderr:tar: Unexpected EOF in archive 2024-09-20T20:20:28.098 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls: Cannot utime: Input/output error 2024-09-20T20:20:28.098 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/kernel: Cannot utime: Input/output error 2024-09-20T20:20:28.098 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2024-09-20T20:20:28.099 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-09-20T20:20:28.099 INFO:tasks.workunit.client.0.smithi114.stderr:tar: Error is not recoverable: exiting now 2024-09-20T20:20:28.099 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-09-20T20:20:28.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-20T20:20:28.142 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-20T20:20:28.143 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-20T20:20:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:27.916+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:27.916+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:27.916+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:27.917+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:28.161+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:28.161+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:28.161+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:28.162+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:28.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:28.365+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:28.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:28.365+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:28.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:28.365+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:28.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:28.365+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:28.943+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:28.943+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:28.943+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:28.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:28.944+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:29.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:29.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:29.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:29.352+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:29.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:29.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:29.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:29.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:29.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:29.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:29.189+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:30.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:29.982+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:30.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:29.982+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:30.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:29.982+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:30.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:29.983+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:30.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:30.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:30.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:30.155+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:30.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:30.388+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:30.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:30.388+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:30.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:30.388+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:30.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:30.388+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:31.366 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:31.027+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:31.366 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:31.027+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:31.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:31.027+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:31.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:31.028+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:31.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:31.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:31.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:31.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:31.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:31.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:31.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:31.175+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:31.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:20:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:20:31.604+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 34087 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:20:31.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:31.364+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:31.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:31.364+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:31.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:31.364+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:31.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:31.364+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:32.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:32.150+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:32.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:32.150+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:32.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:32.150+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:32.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:32.151+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:32.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:32.074+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:32.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:32.074+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:32.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:32.074+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:32.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:32.074+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:32.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:32.376+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:32.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:32.376+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:32.654 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:32.376+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:32.654 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:32.377+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:33.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:33.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:33.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:33.039+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:33.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:33.127+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:33.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:33.127+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:33.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:33.127+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:33.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:33.128+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:33.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:33.376+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:33.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:33.376+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:33.651 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:33.376+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:33.651 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:33.376+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:34.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:34.149+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:34.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:34.149+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:34.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:34.149+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:34.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:34.149+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:34.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:34.064+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:34.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:34.064+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:34.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:34.064+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:34.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:34.064+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:34.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:34.374+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:34.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:34.374+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:34.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:34.374+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:34.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:34.375+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:35.418 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:35.055+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:35.418 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:35.055+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:35.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:35.055+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:35.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:35.055+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:35.419 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:35.167+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:35.419 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:35.167+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:35.419 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:35.167+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:35.419 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:35.167+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:35.415+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:35.415+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:35.415+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:35.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:35.416+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:36.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:36.037+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:36.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:36.037+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:36.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:36.037+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:36.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:36.037+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:36.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:36.186+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:36.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:36.186+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:36.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:36.186+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:36.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:36.187+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:36.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:20:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:20:36.608+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 34176 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:20:36.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:36.377+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:36.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:36.377+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:36.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:36.377+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:36.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:36.378+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:37.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:37.081+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:37.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:37.081+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:37.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:37.081+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:37.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:37.082+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:37.413+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:37.413+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:37.413+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:37.413+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:37.189+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:37.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:37.189+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:37.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:37.189+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:37.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:37.190+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:38.455 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:38.075+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:38.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:38.075+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:38.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:38.075+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:38.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:38.076+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:38.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:38.165+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:38.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:38.165+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:38.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:38.165+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:38.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:38.165+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:38.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:38.453+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:38.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:38.453+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:38.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:38.453+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:38.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:38.454+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:39.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:39.088+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:39.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:39.088+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:39.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:39.088+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:39.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:39.089+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:39.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:39.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:39.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:39.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:39.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:39.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:39.143+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:39.832 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:39.485+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:39.832 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:39.485+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:39.832 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:39.485+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:39.832 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:39.485+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:40.108+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:40.108+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:40.108+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:40.109+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:40.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:40.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:40.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:40.106+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:40.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:40.476+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:40.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:40.476+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:40.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:40.476+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:40.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:40.477+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:41.092+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:41.092+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:41.092+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:41.093+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:41.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:41.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:41.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:41.102+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:41.759 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:20:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:20:41.613+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 34267 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:20:41.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:41.481+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:41.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:41.481+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:41.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:41.481+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:41.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:41.482+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:42.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:42.111+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:42.454 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:42.111+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:42.454 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:42.111+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:42.454 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:42.112+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:42.454 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:42.133+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:42.454 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:42.133+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:42.454 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:42.133+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:42.455 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:42.134+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:42.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:42.451+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:42.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:42.451+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:42.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:42.451+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:42.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:42.452+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:43.416 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:43.184+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:43.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:43.184+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:43.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:43.184+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:43.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:43.185+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:43.417 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:43.117+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:43.417 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:43.117+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:43.417 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:43.117+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:43.418 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:43.118+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:43.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:43.414+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:43.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:43.414+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:43.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:43.414+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:43.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:43.415+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:44.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:44.180+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:44.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:44.180+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:44.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:44.180+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:44.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:44.180+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:44.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:44.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:44.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:44.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:44.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:44.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:44.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:44.107+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:44.423+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:44.423+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:44.423+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:44.424+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:45.399 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:45.168+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:45.399 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:45.168+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:45.399 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:45.168+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:45.399 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:45.168+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:45.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:45.091+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:45.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:45.091+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:45.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:45.091+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:45.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:45.091+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:45.394+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:45.394+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:45.394+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:45.394+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:46.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:46.146+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:46.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:46.146+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:46.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:46.146+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:46.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:46.147+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:46.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:46.122+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:46.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:46.123+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:46.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:46.123+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:46.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:46.123+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:46.631 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:20:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:20:46.617+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 34357 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:20:46.631 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:46.374+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:46.631 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:46.374+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:46.631 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:46.374+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:46.632 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:46.375+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:47.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:47.173+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:47.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:47.173+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:47.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:47.174+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:47.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:47.174+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:47.413 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:47.167+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:47.414 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:47.167+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:47.414 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:47.167+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:47.414 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:47.168+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:47.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:47.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:47.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:47.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:47.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:47.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:47.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:47.411+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:48.400 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:48.171+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:48.401 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:48.171+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:48.401 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:48.171+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:48.401 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:48.171+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:48.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:48.141+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:48.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:48.141+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:48.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:48.141+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:48.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:48.141+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:48.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:48.398+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:48.687 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:48.398+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:48.687 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:48.398+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:48.687 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:48.399+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:49.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:49.169+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:49.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:49.169+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:49.443 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:49.169+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:49.443 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:49.170+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:49.443 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:49.167+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:49.443 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:49.167+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:49.443 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:49.167+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:49.443 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:49.168+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:49.439+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:49.439+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:49.439+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:49.440+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:50.448 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:50.126+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:50.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:50.126+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:50.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:50.126+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:50.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:50.127+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:50.449 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:50.216+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:50.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:50.216+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:50.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:50.216+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:50.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:50.217+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:50.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:50.475+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:50.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:50.475+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:50.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:50.475+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:50.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:50.476+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:51.445 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:51.103+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:51.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:51.103+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:51.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:51.104+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:51.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:51.104+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:51.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:51.204+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:51.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:51.204+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:51.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:51.204+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:51.447 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:51.205+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:51.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:20:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:20:51.621+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 34444 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:20:51.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:51.443+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:51.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:51.443+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:51.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:51.443+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:51.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:51.443+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:52.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:52.098+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:52.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:52.098+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:52.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:52.098+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:52.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:52.099+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:52.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:52.189+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:52.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:52.189+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:52.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:52.189+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:52.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:52.189+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:52.489+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:52.489+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:52.489+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:52.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:52.490+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:53.230+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:53.230+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:53.230+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:53.230+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:53.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:53.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:53.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:53.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:53.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:53.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:53.064+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:53.492+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:53.492+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:53.492+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:53.492+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:54.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:54.223+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:54.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:54.223+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:54.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:54.223+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:54.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:54.224+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:54.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:54.111+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:54.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:54.111+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:54.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:54.111+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:54.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:54.112+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:54.531+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:54.531+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:54.531+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:54.532+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:55.349 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:55.083+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:55.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:55.083+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:55.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:55.083+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:55.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:55.084+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:55.350 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:55.181+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:55.350 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:55.181+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:55.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:55.181+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:55.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:55.182+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:55.527+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:55.527+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:55.527+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:55.528+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:56.097+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:56.097+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:56.097+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:56.098+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:56.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:56.149+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:56.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:56.149+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:56.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:56.149+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:56.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:56.150+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:56.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:20:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:20:56.626+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 34529 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:20:56.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:56.525+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:56.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:56.525+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:56.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:56.525+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:56.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:56.526+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:57.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:57.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:57.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:57.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:57.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:57.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:57.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:57.110+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:57.190+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:57.190+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:57.190+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:57.190+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:57.501+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:57.501+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:57.501+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:57.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:57.502+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:58.223+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:58.223+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:58.223+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:58.224+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:58.105+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:58.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:58.105+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:58.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:58.105+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:58.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:58.106+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:58.548+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:58.548+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:58.548+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:58.548+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:59.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:59.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:59.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:20:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:20:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:20:59.184+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:20:59.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:59.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:59.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:59.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:59.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:59.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:20:59.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:20:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:20:59.063+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:20:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:59.524+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:59.524+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:59.524+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:20:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:20:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:20:59.524+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:00.423 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:00.141+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:00.423 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:00.141+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:00.423 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:00.141+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:00.423 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:00.142+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:00.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:00.072+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:00.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:00.072+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:00.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:00.072+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:00.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:00.073+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:00.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:00.562+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:00.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:00.562+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:00.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:00.562+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:00.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:00.562+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:01.094+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:01.094+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:01.094+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:01.095+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:01.094+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:01.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:01.094+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:01.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:01.094+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:01.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:01.095+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:01.857 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:21:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:21:01.630+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 34620 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:21:01.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:01.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:01.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:01.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:01.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:01.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:01.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:01.562+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:02.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:02.074+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:02.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:02.074+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:02.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:02.074+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:02.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:02.075+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:02.140+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:02.140+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:02.140+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:02.141+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:02.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:02.546+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:02.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:02.546+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:02.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:02.546+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:02.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:02.546+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:03.120+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:03.120+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:03.120+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:03.120+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:03.074+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:03.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:03.074+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:03.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:03.074+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:03.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:03.075+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:03.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:03.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:03.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:03.574+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:04.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:04.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:04.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:04.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:04.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:04.118+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:04.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:04.119+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:04.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:04.072+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:04.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:04.072+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:04.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:04.072+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:04.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:04.072+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:04.622+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:04.622+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:04.622+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:04.623+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:05.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:05.116+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:05.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:05.116+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:05.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:05.116+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:05.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:05.117+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:05.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:05.082+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:05.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:05.082+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:05.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:05.082+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:05.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:05.083+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:05.808 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:05.606+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:05.808 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:05.606+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:05.809 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:05.606+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:05.809 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:05.606+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:06.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:06.160+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:06.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:06.160+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:06.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:06.160+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:06.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:06.161+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:06.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:06.132+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:06.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:06.132+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:06.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:06.132+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:06.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:06.133+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:06.868 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:21:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:21:06.634+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 34710 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:21:06.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:06.594+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:06.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:06.594+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:06.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:06.594+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:06.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:06.595+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:07.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:07.142+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:07.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:07.142+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:07.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:07.142+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:07.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:07.143+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:07.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:07.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:07.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:07.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:07.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:07.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:07.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:07.144+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:07.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:07.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:07.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:07.576+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:08.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:08.136+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:08.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:08.136+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:08.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:08.136+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:08.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:08.137+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:08.149+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:08.149+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:08.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:08.149+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:08.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:08.150+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:08.566+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:08.566+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:08.566+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:08.567+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:09.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:09.177+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:09.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:09.177+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:09.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:09.177+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:09.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:09.178+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:09.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:09.113+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:09.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:09.113+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:09.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:09.113+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:09.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:09.113+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:09.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:09.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:09.571+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:09.572+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:10.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:10.203+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:10.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:10.203+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:10.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:10.203+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:10.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:10.204+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:10.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:10.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:10.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:10.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:10.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:10.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:10.154+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:10.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:10.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:10.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:10.587+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:11.185+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:11.185+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:11.185+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:11.185+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:11.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:11.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:11.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:11.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:11.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:11.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:11.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:11.188+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:11.952 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:21:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:21:11.640+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 34800 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:21:11.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:11.637+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:11.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:11.637+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:11.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:11.637+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:11.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:11.638+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:12.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:12.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:12.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:12.148+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:12.685 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:12.232+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:12.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:12.232+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:12.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:12.232+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:12.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:12.232+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:12.683+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:12.683+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:12.683+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:12.684+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:13.232+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:13.232+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:13.232+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:13.232+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:13.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:13.104+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:13.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:13.104+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:13.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:13.104+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:13.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:13.105+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:13.696+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:13.696+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:13.696+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:13.697+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:14.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:14.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:14.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:14.276+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:14.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:14.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:14.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:14.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:14.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:14.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:14.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:14.106+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:15.024 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:14.740+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:15.024 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:14.740+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:15.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:14.740+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:15.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:14.741+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:15.279 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:15.129+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:15.279 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:15.129+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:15.279 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:15.129+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:15.279 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:15.130+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:15.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:15.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:15.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:15.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:15.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:15.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:15.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:15.277+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:15.784+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:15.784+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:15.784+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:15.785+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:16.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:16.144+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:16.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:16.144+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:16.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:16.144+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:16.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:16.145+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:16.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:16.309+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:16.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:16.309+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:16.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:16.309+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:16.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:16.309+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:16.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:21:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:21:16.645+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 34880 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:21:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:16.799+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:16.799+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:16.799+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:16.800+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:17.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:17.189+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:17.311 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:17.189+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:17.311 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:17.189+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:17.311 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:17.191+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:17.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:17.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:17.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:17.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:17.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:17.309+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:18.213 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:17.829+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:18.213 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:17.829+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:18.213 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:17.829+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:18.213 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:17.830+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:18.287+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:18.287+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:18.287+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:18.287+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:18.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:18.210+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:18.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:18.210+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:18.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:18.210+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:18.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:18.211+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:18.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:19.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:18.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:19.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:18.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:19.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:18.833+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:19.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:19.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:19.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:19.285+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:19.242+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:19.242+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:19.242+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:19.243+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:19.851+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:19.851+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:19.851+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:19.852+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:20.305+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:20.305+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:20.305+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:20.305+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:20.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:20.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:20.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:20.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:20.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:20.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:20.283+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:21.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:20.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:21.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:20.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:21.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:20.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:21.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:20.857+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:21.303+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:21.303+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:21.303+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:21.303+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:21.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:21.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:21.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:21.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:21.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:21.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:21.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:21.265+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:21.907 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:21:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:21:21.649+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 34970 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:21:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:21.905+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:21.905+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:21.905+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:21.905+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:22.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:22.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:22.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:22.297+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:22.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:22.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:22.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:22.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:22.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:22.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:22.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:22.265+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:22.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:22.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:22.871+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:22.871+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:23.653 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:23.292+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:23.653 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:23.292+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:23.653 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:23.292+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:23.653 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:23.292+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:23.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:23.261+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:23.654 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:23.261+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:23.654 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:23.261+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:23.654 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:23.262+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:23.912+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:23.912+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:23.912+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:23.913+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:24.292+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:24.292+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:24.292+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:24.293+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:24.276+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:24.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:24.276+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:24.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:24.276+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:24.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:24.276+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:24.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:24.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:24.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:24.916+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:25.254+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:25.254+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:25.254+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:25.255+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:25.249+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:25.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:25.249+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:25.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:25.249+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:25.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:25.250+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:25.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:25.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:25.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:25.911+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:26.656 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:26.220+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:26.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:26.220+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:26.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:26.220+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:26.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:26.221+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:26.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:26.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:26.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:26.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:26.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:26.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:26.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:26.237+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:26.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:21:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:21:26.654+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 35060 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:21:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:26.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:26.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:26.873+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:26.874+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:27.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:27.188+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:27.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:27.188+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:27.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:27.188+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:27.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:27.189+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:27.253+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:27.253+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:27.253+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:27.254+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:28.189 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:27.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:28.190 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:27.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:28.190 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:27.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:28.190 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:27.843+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:28.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:28.187+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:28.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:28.187+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:28.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:28.187+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:28.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:28.188+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:28.284+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:28.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:28.284+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:28.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:28.284+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:28.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:28.285+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:29.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:28.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:29.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:28.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:29.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:28.816+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:29.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:28.817+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:29.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:29.156+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:29.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:29.156+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:29.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:29.156+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:29.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:29.157+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:29.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:29.260+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:29.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:29.260+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:29.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:29.260+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:29.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:29.261+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:30.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:29.775+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:30.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:29.775+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:30.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:29.775+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:30.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:29.775+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:30.384 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:30.108+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:30.385 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:30.108+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:30.385 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:30.108+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:30.385 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:30.109+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:30.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:30.231+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:30.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:30.231+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:30.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:30.231+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:30.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:30.232+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:31.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:30.802+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:31.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:30.802+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:31.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:30.802+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:31.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:30.803+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:31.156+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:31.156+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:31.156+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:31.156+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:31.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:31.198+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:31.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:31.198+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:31.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:31.198+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:31.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:31.199+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:31.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:21:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:21:31.658+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 35151 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:21:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:31.761+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:31.761+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:31.761+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:31.762+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:32.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:32.111+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:32.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:32.111+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:32.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:32.111+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:32.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:32.111+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:32.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:32.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:32.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:32.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:32.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:32.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:32.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:32.176+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:33.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:32.737+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:33.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:32.737+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:33.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:32.737+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:33.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:32.738+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:33.130+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:33.130+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:33.130+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:33.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:33.131+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:33.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:33.110+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:33.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:33.110+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:33.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:33.110+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:33.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:33.111+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:33.743+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:33.743+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:33.743+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:33.744+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:34.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:34.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:34.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:34.110+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:34.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:34.177+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:34.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:34.177+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:34.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:34.177+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:34.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:34.178+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:35.066 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:34.726+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:35.066 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:34.726+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:35.066 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:34.726+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:35.066 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:34.727+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:35.222+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:35.222+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:35.222+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:35.223+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:35.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:35.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:35.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:35.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:35.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:35.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:35.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:35.064+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:36.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:35.741+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:36.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:35.741+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:36.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:35.741+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:36.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:35.741+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:36.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:36.083+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:36.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:36.083+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:36.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:36.083+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:36.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:36.084+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:36.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:36.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:36.529 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:36.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:36.529 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:36.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:36.529 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:36.273+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:36.869 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:21:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:21:36.663+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 35241 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:21:36.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:36.763+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:36.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:36.763+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:36.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:36.763+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:36.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:36.764+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:37.076+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:37.076+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:37.076+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:37.077+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:37.259+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:37.259+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:37.259+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:37.259+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:38.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:37.792+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:38.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:37.792+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:38.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:37.792+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:38.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:37.793+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:38.116+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:38.116+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:38.116+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:38.117+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:38.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:38.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:38.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:38.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:38.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:38.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:38.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:38.252+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:39.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:38.747+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:39.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:38.747+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:39.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:38.747+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:39.123 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:38.748+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:39.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:39.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:39.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:39.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:39.237+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:39.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:39.119+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:39.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:39.119+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:39.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:39.119+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:39.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:39.120+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:40.139 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:39.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:40.139 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:39.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:40.139 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:39.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:40.139 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:39.753+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:40.136+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:40.136+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:40.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:40.136+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:40.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:40.137+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:40.270+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:40.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:40.270+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:40.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:40.270+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:40.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:40.271+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:41.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:40.792+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:41.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:40.792+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:41.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:40.792+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:41.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:40.793+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:41.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:41.275+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:41.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:41.275+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:41.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:41.275+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:41.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:41.275+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:41.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:41.137+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:41.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:41.137+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:41.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:41.137+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:41.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:41.138+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:41.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:21:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:21:41.667+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 35331 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:21:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:41.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:41.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:41.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:41.819+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:42.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:42.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:42.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:42.228+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:42.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:42.157+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:42.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:42.157+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:42.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:42.157+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:42.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:42.158+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:42.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:42.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:42.835+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:42.835+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:43.153+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:43.153+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:43.153+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:43.153+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:43.271+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:43.271+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:43.271+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:43.272+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:43.790+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:43.790+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:43.790+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:43.791+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:44.248+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:44.248+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:44.248+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:44.249+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:44.176+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:44.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:44.176+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:44.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:44.176+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:44.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:44.176+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:45.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:44.815+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:45.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:44.815+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:45.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:44.815+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:45.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:44.816+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:45.246+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:45.246+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:45.246+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:45.247+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:45.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:45.172+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:45.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:45.172+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:45.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:45.172+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:45.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:45.173+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:46.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:45.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:46.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:45.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:46.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:45.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:46.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:45.842+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:46.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:46.280+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:46.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:46.280+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:46.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:46.280+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:46.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:46.281+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:46.190+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:46.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:46.190+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:46.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:46.190+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:46.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:46.191+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:46.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:21:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:21:46.672+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 35422 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:21:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:46.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:46.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:46.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:46.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:46.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:46.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:46.838+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:47.309 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:47.200+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:47.309 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:47.200+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:47.310 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:47.200+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:47.310 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:47.201+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:47.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:47.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:47.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:47.308+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:48.190 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:47.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:48.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:47.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:48.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:47.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:48.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:47.876+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:48.188+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:48.188+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:48.188+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:48.188+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:48.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:48.354+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:48.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:48.354+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:48.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:48.354+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:48.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:48.355+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:49.149 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:48.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:49.149 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:48.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:49.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:48.881+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:49.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:48.882+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:49.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:49.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:49.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:49.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:49.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:49.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:49.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:49.322+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:49.147+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:49.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:49.147+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:49.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:49.147+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:49.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:49.148+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:50.138 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:49.847+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:50.138 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:49.847+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:50.139 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:49.847+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:50.139 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:49.848+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:50.296+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:50.296+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:50.296+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:50.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:50.297+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:50.135+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:50.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:50.135+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:50.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:50.135+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:50.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:50.136+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:51.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:50.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:51.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:50.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:51.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:50.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:51.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:50.865+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:51.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:51.274+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:51.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:51.274+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:51.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:51.274+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:51.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:51.274+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:51.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:51.121+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:51.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:51.121+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:51.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:51.121+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:51.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:51.122+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:51.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:21:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:21:51.677+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 35512 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:21:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:51.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:51.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:51.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:51.820+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:52.314+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:52.314+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:52.314+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:52.315+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:52.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:52.152+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:52.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:52.152+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:52.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:52.152+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:52.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:52.153+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:53.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:52.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:53.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:52.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:53.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:52.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:53.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:52.868+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:53.150+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:53.150+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:53.150+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:53.150+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:53.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:53.341+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:53.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:53.341+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:53.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:53.341+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:53.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:53.342+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:53.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:53.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:53.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:53.820+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:54.122+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:54.122+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:54.122+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:54.123+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:54.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:54.320+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:54.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:54.320+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:54.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:54.320+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:54.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:54.321+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:55.106 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:54.807+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:55.106 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:54.807+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:55.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:54.807+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:55.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:54.808+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:55.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:55.103+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:55.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:55.103+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:55.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:55.103+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:55.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:55.104+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:55.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:55.294+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:55.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:55.294+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:55.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:55.294+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:55.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:55.295+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:56.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:55.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:56.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:55.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:56.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:55.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:56.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:55.829+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:56.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:56.289+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:56.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:56.289+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:56.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:56.289+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:56.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:56.289+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:56.148+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:56.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:56.148+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:56.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:56.148+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:56.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:56.148+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:56.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:21:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:21:56.681+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 35602 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:21:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:56.792+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:56.792+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:56.792+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:56.793+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:57.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:57.102+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:57.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:57.102+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:57.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:57.102+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:57.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:57.103+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:57.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:57.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:57.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:57.265+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:58.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:57.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:58.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:57.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:58.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:57.842+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:58.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:57.843+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:58.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:58.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:58.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:58.110+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:58.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:58.312+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:58.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:58.312+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:58.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:58.312+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:58.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:58.313+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:59.108 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:58.840+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:59.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:58.840+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:59.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:58.840+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:21:59.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:58.840+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:21:59.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:59.105+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:59.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:59.105+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:59.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:59.105+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:21:59.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:21:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:21:59.106+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:21:59.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:59.276+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:59.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:59.276+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:59.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:59.276+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:21:59.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:21:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:21:59.277+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:00.086+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:00.086+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:00.086+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:00.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:00.086+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:00.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:59.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:00.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:59.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:00.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:59.876+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:00.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:21:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:21:59.877+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:00.255+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:00.255+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:00.255+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:00.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:00.256+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:01.085 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:00.858+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:01.085 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:00.858+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:01.085 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:00.858+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:01.086 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:00.859+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:01.393 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:01.082+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:01.393 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:01.082+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:01.393 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:01.082+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:01.393 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:01.083+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:01.394 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:01.268+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:01.394 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:01.268+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:01.394 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:01.268+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:01.394 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:01.269+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:01.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:22:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:22:01.686+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 35692 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:22:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:01.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:01.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:01.830+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:01.831+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:02.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:02.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:02.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:02.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:02.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:02.237+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:02.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:02.034+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:02.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:02.034+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:02.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:02.034+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:02.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:02.034+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:03.044+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:03.044+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:03.044+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:03.044+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:02.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:03.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:02.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:03.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:02.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:03.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:02.881+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:03.656 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:03.250+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:03.656 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:03.250+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:03.656 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:03.250+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:03.656 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:03.251+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:04.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:04.044+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:04.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:04.044+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:04.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:04.044+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:04.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:04.045+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:04.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:03.912+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:04.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:03.912+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:04.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:03.912+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:04.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:03.913+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:04.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:04.263+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:04.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:04.263+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:04.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:04.263+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:04.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:04.264+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:04.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:04.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:04.866+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:04.867+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:05.065+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:05.065+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:05.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:05.065+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:05.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:05.066+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:05.257+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:05.257+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:05.257+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:05.257+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:06.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:06.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:06.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:06.063+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:05.848+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:05.848+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:05.848+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:05.849+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:06.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:06.231+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:06.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:06.231+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:06.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:06.231+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:06.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:06.232+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:06.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:22:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:22:06.691+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 35783 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:22:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:06.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:06.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:06.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:06.869+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:07.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:07.069+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:07.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:07.069+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:07.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:07.069+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:07.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:07.069+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:07.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:07.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:07.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:07.252+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:08.060+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:08.060+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:08.060+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:08.060+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:07.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:08.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:07.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:08.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:07.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:08.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:07.917+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:08.301+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:08.301+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:08.301+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:08.302+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:09.013+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:09.013+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:09.013+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:09.013+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:08.877+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:08.877+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:09.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:08.877+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:09.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:08.878+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:09.257+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:09.257+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:09.257+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:09.257+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:10.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:10.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:10.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:10.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:10.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:10.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:10.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:10.022+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:09.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:09.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:09.850+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:09.851+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:10.212+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:10.212+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:10.212+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:10.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:10.213+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:10.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:10.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:10.828+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:11.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:10.829+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:11.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:11.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:11.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:11.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:11.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:11.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:11.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:11.001+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:11.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:11.231+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:11.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:11.231+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:11.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:11.231+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:11.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:11.232+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:11.839 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:22:11.696+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 35873 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:22:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:11.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:11.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:11.837+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:11.837+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:12.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:11.986+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:12.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:11.986+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:12.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:11.986+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:12.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:11.987+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:12.270+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:12.270+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:12.270+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:12.270+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:12.964+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:12.964+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:12.964+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:12.965+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:13.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:12.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:13.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:12.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:13.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:12.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:13.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:12.834+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:13.656 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:13.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:13.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:13.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:13.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:13.251+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:13.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:13.252+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:13.813+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:13.813+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:13.813+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:13.814+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:14.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:13.975+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:14.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:13.975+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:14.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:13.975+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:14.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:13.976+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:14.258+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:14.258+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:14.258+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:14.258+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:14.989+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:14.989+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:14.989+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:14.990+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:14.800+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:15.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:14.800+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:15.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:14.800+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:15.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:14.800+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:15.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:15.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:15.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:15.246+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:15.778+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:15.778+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:15.778+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:16.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:15.779+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:16.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:15.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:16.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:15.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:16.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:15.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:16.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:15.972+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:16.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:16.249+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:16.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:16.249+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:16.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:16.249+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:16.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:16.250+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:16.953 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:22:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:22:16.701+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 35963 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:22:16.954 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:16.744+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:16.954 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:16.744+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:16.954 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:16.744+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:16.954 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:16.745+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:16.951+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:16.951+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:16.951+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:16.952+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:17.224+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:17.224+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:17.224+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:17.225+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:17.747+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:17.747+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:17.747+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:17.747+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:17.961+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:17.961+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:18.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:17.961+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:18.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:17.962+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:18.217+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:18.217+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:18.217+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:18.218+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:19.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:18.929+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:19.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:18.929+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:19.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:18.929+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:19.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:18.930+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:19.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:18.739+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:19.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:18.739+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:19.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:18.739+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:19.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:18.740+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:19.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:19.172+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:19.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:19.172+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:19.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:19.172+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:19.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:19.173+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:20.214 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:19.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:20.214 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:19.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:20.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:19.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:20.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:19.888+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:20.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:19.788+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:20.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:19.788+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:20.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:19.788+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:20.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:19.789+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:20.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:20.212+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:20.212+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:20.212+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:20.212+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:21.211 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:20.794+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:21.212 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:20.794+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:21.212 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:20.794+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:21.212 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:20.794+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:21.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:20.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:21.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:20.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:21.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:20.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:21.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:20.882+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:21.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:21.209+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:21.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:21.209+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:21.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:21.209+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:21.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:21.210+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:21.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:22:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:22:21.705+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 36053 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:22:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:21.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:21.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:21.848+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:21.848+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:21.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:21.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:21.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:21.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:21.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:21.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:21.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:21.748+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:22.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:22.179+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:22.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:22.179+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:22.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:22.179+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:22.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:22.180+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:22.885 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:22.761+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:22.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:22.761+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:22.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:22.761+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:22.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:22.761+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:23.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:22.883+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:23.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:22.883+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:23.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:22.883+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:23.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:22.884+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:23.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:23.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:23.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:23.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:23.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:23.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:23.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:23.229+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:23.920 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:23.797+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:23.921 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:23.797+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:23.921 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:23.797+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:23.921 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:23.798+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:24.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:23.918+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:24.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:23.918+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:24.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:23.918+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:24.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:23.919+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:24.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:24.220+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:24.220+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:24.220+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:24.220+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:24.846+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:24.846+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:24.846+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:24.847+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:25.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:24.914+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:25.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:24.914+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:25.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:24.914+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:25.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:24.914+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:25.681 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:25.263+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:25.682 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:25.263+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:25.682 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:25.263+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:25.682 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:25.263+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:25.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:25.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:25.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:25.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:25.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:25.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:25.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:25.881+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:25.932+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:25.932+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:25.932+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:25.933+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:26.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:26.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:26.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:26.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:26.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:26.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:26.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:26.265+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:26.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:22:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:22:26.710+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 36144 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:22:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:26.857+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:26.858+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:26.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:26.858+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:26.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:26.858+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:27.263 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:26.978+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:27.263 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:26.978+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:27.263 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:26.978+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:27.263 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:26.979+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:27.260+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:27.260+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:27.260+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:27.261+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:27.984+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:27.984+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:27.984+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:27.985+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:27.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:27.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:27.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:27.900+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:28.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:28.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:28.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:28.221+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:29.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:28.986+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:29.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:28.986+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:29.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:28.986+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:29.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:28.986+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:29.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:28.894+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:29.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:28.894+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:29.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:28.894+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:29.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:28.894+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:29.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:29.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:29.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:29.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:29.204+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:30.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:29.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:30.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:29.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:30.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:29.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:30.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:29.938+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:30.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:29.883+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:30.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:29.883+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:30.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:29.883+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:30.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:29.883+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:30.165+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:30.165+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:30.165+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:30.165+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:31.196 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:30.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:31.196 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:30.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:31.196 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:30.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:31.196 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:30.916+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:31.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:30.917+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:31.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:30.917+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:31.197 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:30.917+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:31.197 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:30.918+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:31.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:31.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:31.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:31.194+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:32.209 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:22:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:22:31.715+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 36234 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:22:32.209 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:31.876+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:32.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:31.876+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:32.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:31.876+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:32.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:31.877+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:32.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:31.938+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:32.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:31.938+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:32.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:31.938+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:32.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:31.939+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:32.206+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:32.206+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:32.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:32.206+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:32.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:32.207+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:32.977+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:32.977+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:32.977+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:33.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:32.978+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:33.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:32.876+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:33.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:32.876+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:33.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:32.876+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:33.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:32.876+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:33.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:33.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:33.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:33.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:33.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:33.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:33.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:33.245+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:33.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:33.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:33.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:33.879+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:34.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:33.986+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:34.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:33.986+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:34.249 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:33.986+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:34.249 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:33.987+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:34.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:34.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:34.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:34.246+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:34.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:34.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:34.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:34.903+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:34.994+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:34.994+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:35.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:34.994+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:35.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:34.995+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:35.252+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:35.252+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:35.252+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:35.253+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:35.916+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:35.916+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:35.916+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:35.916+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:35.990+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:35.990+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:35.990+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:35.990+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:36.238+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:36.238+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:36.238+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:36.239+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:36.974 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:22:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:22:36.720+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 36324 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:22:36.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:36.912+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:36.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:36.912+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:36.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:36.912+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:36.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:36.912+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:37.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:37.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:37.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:37.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:37.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:37.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:37.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:37.036+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:37.274+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:37.274+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:37.274+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:37.275+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:38.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:38.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:38.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:38.085+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:37.900+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:38.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:37.900+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:38.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:37.900+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:38.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:37.901+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:38.267+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:38.267+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:38.267+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:38.267+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:39.057+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:39.057+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:39.057+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:39.058+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:38.856+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:39.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:38.856+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:39.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:38.856+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:39.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:38.857+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:39.292+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:39.292+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:39.292+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:39.292+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:40.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:40.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:40.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:40.084+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:39.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:39.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:40.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:39.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:40.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:39.880+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:40.285+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:40.285+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:40.285+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:40.286+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:41.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:40.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:41.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:40.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:41.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:40.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:41.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:40.862+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:41.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:41.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:41.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:41.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:41.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:41.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:41.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:41.113+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:41.292 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:41.287+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:41.293 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:41.287+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:41.293 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:41.287+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:41.287+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:42.065 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:22:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:22:41.724+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 36414 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:22:42.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:41.867+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:42.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:41.867+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:42.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:41.867+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:42.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:41.868+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:42.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:42.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:42.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:42.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:42.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:42.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:42.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:42.077+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:42.322+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:42.322+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:42.322+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:42.322+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:43.090+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:43.090+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:43.090+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:43.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:43.092+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:42.899+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:43.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:42.899+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:43.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:42.899+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:43.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:42.900+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:43.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:43.363+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:43.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:43.363+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:43.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:43.363+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:43.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:43.364+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:44.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:43.909+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:44.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:43.909+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:44.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:43.909+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:44.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:43.910+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:44.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:44.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:44.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:44.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:44.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:44.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:44.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:44.088+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:44.330+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:44.330+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:44.330+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:44.331+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:45.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:44.925+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:45.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:44.925+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:45.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:44.925+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:45.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:44.926+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:45.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:45.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:45.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:45.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:45.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:45.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:45.116+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:45.317+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:45.317+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:45.317+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:45.317+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:45.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:45.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:45.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:45.949+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:46.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:46.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:46.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:46.110+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:46.361+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:46.361+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:46.361+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:46.361+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:47.090 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:22:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:22:46.729+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 36505 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:22:47.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:46.928+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:47.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:46.928+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:47.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:46.928+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:47.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:46.929+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:47.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:47.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:47.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:47.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:47.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:47.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:47.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:47.088+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:47.359+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:47.359+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:47.359+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:47.360+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:48.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:47.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:48.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:47.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:48.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:47.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:48.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:47.973+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:48.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:48.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:48.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:48.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:48.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:48.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:48.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:48.110+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:48.333+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:48.333+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:48.333+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:48.334+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:49.349 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:49.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:49.349 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:49.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:49.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:49.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:49.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:49.010+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:49.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:49.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:49.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:49.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:49.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:49.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:49.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:49.114+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:49.347+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:49.347+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:49.347+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:49.347+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:50.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:50.055+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:50.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:50.055+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:50.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:50.055+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:50.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:50.056+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:50.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:50.082+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:50.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:50.082+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:50.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:50.082+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:50.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:50.083+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:50.361+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:50.361+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:50.361+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:50.362+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:51.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:51.008+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:51.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:51.008+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:51.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:51.008+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:51.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:51.008+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:51.375 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:51.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:51.375 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:51.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:51.375 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:51.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:51.376 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:51.037+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:51.372+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:51.372+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:51.372+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:51.373+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:51.991 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:22:51.734+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 36595 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:22:52.332 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:52.014+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:52.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:52.014+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:52.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:52.014+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:52.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:52.015+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:52.333 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:51.989+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:52.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:51.989+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:52.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:51.989+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:52.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:51.990+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:52.330+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:52.330+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:52.330+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:52.331+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:52.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:52.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:52.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:52.954+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:53.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:53.050+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:53.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:53.050+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:53.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:53.050+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:53.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:53.051+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:53.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:53.351+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:53.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:53.351+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:53.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:53.351+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:53.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:53.351+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:54.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:54.047+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:54.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:54.047+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:54.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:54.047+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:54.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:54.048+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:54.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:53.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:54.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:53.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:54.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:53.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:54.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:53.929+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:54.341+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:54.341+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:54.341+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:54.341+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:55.058+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:55.058+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:55.058+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:55.058+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:54.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:54.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:54.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:54.907+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:55.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:55.387+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:55.387+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:55.387+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:55.388+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:56.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:56.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:56.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:56.018+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:55.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:55.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:55.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:55.940+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:56.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:56.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:56.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:56.424+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:57.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:22:56.739+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 36685 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:22:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:56.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:56.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:56.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:56.967+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:57.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:56.981+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:57.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:56.981+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:57.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:56.981+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:57.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:56.982+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:57.376+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:57.376+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:57.376+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:57.377+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:57.943+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:57.943+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:57.943+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:57.944+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:58.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:57.986+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:58.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:57.986+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:58.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:57.986+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:58.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:57.987+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:58.414+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:58.414+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:58.414+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:58.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:58.415+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:59.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:58.929+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:59.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:58.929+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:59.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:58.929+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:22:59.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:58.929+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:22:59.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:58.947+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:59.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:58.947+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:59.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:58.947+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:22:59.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:58.947+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:22:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:59.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:59.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:59.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:22:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:22:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:22:59.415+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:00.241 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:59.970+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:00.241 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:59.970+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:00.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:59.970+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:00.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:22:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:22:59.970+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:00.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:59.976+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:00.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:59.976+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:00.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:59.976+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:00.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:22:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:22:59.976+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:00.404+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:00.404+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:00.404+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:00.405+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:00.955+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:00.955+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:00.955+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:00.955+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:00.931+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:01.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:00.931+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:01.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:00.931+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:01.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:00.932+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:01.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:01.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:01.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:01.437+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:02.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:23:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:23:01.744+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 36775 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:23:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:01.982+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:01.982+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:01.982+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:02.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:01.983+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:02.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:01.925+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:02.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:01.925+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:02.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:01.925+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:02.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:01.926+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:02.545 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:02.451+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:02.546 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:02.451+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:02.546 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:02.451+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:02.546 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:02.452+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:02.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:02.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:02.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:02.973+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:02.929+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:02.929+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:03.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:02.929+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:03.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:02.930+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:03.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:03.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:03.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:03.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:03.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:03.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:03.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:03.491+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:03.970+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:03.970+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:03.970+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:03.971+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:03.910+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:03.910+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:04.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:03.910+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:04.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:03.910+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:04.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:04.502+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:04.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:04.502+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:04.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:04.502+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:04.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:04.503+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:04.946+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:04.946+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:04.946+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:05.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:04.946+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:04.929+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:05.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:04.929+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:05.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:04.929+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:05.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:04.930+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:05.459+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:05.459+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:05.459+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:05.460+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:05.917+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:05.917+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:05.917+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:05.917+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:05.898+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:05.898+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:05.898+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:05.898+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:06.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:23:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:23:06.749+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 36866 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:23:06.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:06.509+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:06.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:06.509+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:06.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:06.509+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:06.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:06.510+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:06.950+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:06.950+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:06.950+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:06.951+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:06.919+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:07.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:06.919+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:07.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:06.919+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:07.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:06.920+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:07.469+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:07.469+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:07.469+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:07.469+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:08.442 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:07.998+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:08.442 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:07.998+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:08.442 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:07.998+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:08.443 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:07.999+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:08.443 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:07.964+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:08.443 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:07.964+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:08.443 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:07.964+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:08.443 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:07.964+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:08.439+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:08.439+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:08.439+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:08.440+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:09.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:08.998+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:09.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:08.998+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:09.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:08.998+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:09.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:08.998+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:09.455 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:08.998+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:09.455 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:08.998+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:09.455 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:08.998+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:09.455 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:08.999+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:09.451+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:09.451+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:09.451+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:09.452+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:10.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:09.976+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:10.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:09.976+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:10.243 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:09.976+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:10.243 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:09.977+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:10.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:10.013+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:10.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:10.013+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:10.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:10.013+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:10.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:10.013+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:10.446+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:10.446+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:10.446+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:10.447+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:11.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:10.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:11.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:10.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:11.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:10.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:11.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:10.998+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:11.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:11.020+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:11.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:11.020+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:11.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:11.020+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:11.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:11.020+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:11.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:11.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:11.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:11.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:11.432+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:12.014 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:23:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:23:11.754+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 36956 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:23:12.361 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:12.012+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:12.361 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:12.012+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:12.361 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:12.012+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:12.361 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:12.012+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:12.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:12.015+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:12.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:12.015+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:12.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:12.015+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:12.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:12.016+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:12.418+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:12.418+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:12.418+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:12.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:12.418+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:13.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:13.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:13.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:13.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:13.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:13.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:13.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:13.060+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:13.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:12.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:13.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:12.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:13.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:12.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:13.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:12.972+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:13.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:13.374+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:13.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:13.374+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:13.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:13.374+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:13.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:13.375+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:13.937+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:13.937+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:13.937+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:13.938+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:14.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:14.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:14.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:14.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:14.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:14.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:14.070+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:14.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:14.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:14.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:14.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:14.423+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:15.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:14.894+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:15.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:14.894+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:15.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:14.894+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:15.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:14.894+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:15.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:15.099+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:15.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:15.099+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:15.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:15.099+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:15.378 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:15.100+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:15.375+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:15.375+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:15.375+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:15.375+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:15.914+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:15.914+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:15.914+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:15.915+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:16.051+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:16.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:16.051+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:16.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:16.051+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:16.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:16.052+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:16.405+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:16.405+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:16.405+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:16.406+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:17.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:23:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:23:16.759+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 37045 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:23:17.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:16.883+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:17.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:16.883+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:17.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:16.883+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:17.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:16.884+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:17.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:17.018+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:17.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:17.018+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:17.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:17.018+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:17.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:17.019+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:17.385+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:17.385+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:17.385+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:17.385+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:17.920+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:17.920+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:17.920+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:17.921+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:18.067+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:18.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:18.067+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:18.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:18.067+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:18.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:18.068+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:18.340+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:18.340+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:18.340+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:18.340+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:18.953+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:18.953+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:18.953+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:18.953+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:19.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:19.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:19.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:19.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:19.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:19.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:19.070+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:19.339+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:19.339+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:19.339+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:19.340+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:20.243 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:19.975+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:20.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:19.975+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:20.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:19.975+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:20.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:19.975+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:20.244 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:20.021+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:20.244 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:20.021+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:20.244 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:20.021+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:20.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:20.021+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:20.386+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:20.386+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:20.386+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:20.386+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:21.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:21.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:21.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:21.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:21.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:21.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:21.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:21.009+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:21.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:21.071+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:21.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:21.071+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:21.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:21.071+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:21.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:21.071+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:21.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:21.362+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:21.362+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:21.362+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:21.363+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:22.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:23:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:23:21.763+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 37134 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:23:22.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:22.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:22.351 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:22.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:22.351 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:22.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:22.351 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:22.031+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:22.351 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:22.106+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:22.351 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:22.106+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:22.351 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:22.106+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:22.351 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:22.106+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:22.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:22.348+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:22.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:22.348+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:22.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:22.348+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:22.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:22.348+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:23.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:23.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:23.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:23.031+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:23.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:23.031+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:23.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:23.031+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:23.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:23.061+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:23.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:23.061+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:23.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:23.061+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:23.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:23.062+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:23.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:23.361+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:23.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:23.361+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:23.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:23.361+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:23.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:23.362+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:24.345 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:23.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:24.346 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:23.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:24.346 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:23.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:24.346 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:23.989+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:24.346 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:24.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:24.346 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:24.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:24.346 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:24.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:24.347 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:24.111+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:24.342+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:24.342+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:24.342+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:24.342+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:25.122+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:25.122+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:25.122+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:25.122+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:25.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:24.952+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:25.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:24.952+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:25.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:24.952+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:25.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:24.953+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:25.375+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:25.375+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:25.375+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:25.376+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:26.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:25.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:26.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:25.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:26.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:25.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:26.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:25.985+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:26.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:26.086+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:26.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:26.086+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:26.378 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:26.086+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:26.378 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:26.086+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:26.373+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:26.373+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:26.373+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:26.374+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:27.076 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:23:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:23:26.767+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 37225 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:23:27.077 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:26.947+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:27.077 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:26.947+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:27.077 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:26.947+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:27.077 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:26.947+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:27.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:27.073+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:27.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:27.074+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:27.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:27.074+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:27.422 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:27.074+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:27.417+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:27.417+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:27.417+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:27.419+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:28.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:27.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:28.043 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:27.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:28.043 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:27.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:28.043 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:27.967+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:28.403 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:28.040+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:28.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:28.040+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:28.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:28.040+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:28.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:28.041+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:28.401+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:28.401+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:28.401+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:28.402+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:29.394 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:29.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:29.394 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:29.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:29.394 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:29.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:29.395 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:29.001+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:29.395 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:29.053+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:29.395 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:29.053+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:29.395 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:29.053+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:29.395 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:29.053+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:29.391+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:29.391+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:29.391+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:29.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:29.392+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:30.243 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:29.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:30.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:29.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:30.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:29.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:30.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:29.974+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:30.244 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:30.018+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:30.244 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:30.018+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:30.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:30.018+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:30.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:30.019+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:30.372+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:30.372+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:30.372+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:30.372+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:31.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:30.994+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:31.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:30.994+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:31.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:30.994+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:31.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:30.994+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:31.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:30.994+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:31.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:30.994+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:31.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:30.994+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:31.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:30.995+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:31.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:31.421+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:31.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:31.421+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:31.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:31.421+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:31.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:31.422+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:32.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:23:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:23:31.773+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 37315 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:23:32.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:31.945+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:32.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:31.945+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:32.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:31.945+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:32.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:31.946+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:32.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:32.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:32.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:32.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:32.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:32.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:32.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:32.038+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:32.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:32.412+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:32.412+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:32.412+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:32.413+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:32.940+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:33.227 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:32.940+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:33.227 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:32.940+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:33.227 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:32.940+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:33.228 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:33.015+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:33.228 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:33.015+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:33.228 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:33.015+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:33.228 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:33.015+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:33.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:33.445+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:33.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:33.445+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:33.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:33.445+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:33.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:33.446+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:33.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:33.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:33.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:33.902+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:34.006+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:34.006+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:34.006+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:34.006+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:34.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:34.474+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:34.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:34.474+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:34.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:34.474+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:34.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:34.475+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:34.910+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:34.910+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:34.910+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:35.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:34.910+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:35.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:35.041+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:35.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:35.041+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:35.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:35.041+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:35.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:35.041+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:35.901 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:35.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:35.901 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:35.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:35.901 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:35.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:35.901 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:35.515+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:35.898+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:35.898+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:35.898+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:35.899+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:36.032+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:36.032+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:36.032+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:36.032+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:36.538+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:36.538+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:36.538+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:36.538+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:37.072 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:23:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:23:36.778+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 37405 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:23:37.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:36.870+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:37.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:36.870+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:37.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:36.870+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:37.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:36.871+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:37.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:37.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:37.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:37.070+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:37.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:37.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:37.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:37.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:37.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:37.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:37.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:37.582+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:37.889+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:37.889+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:37.889+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:37.890+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:38.061+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:38.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:38.061+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:38.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:38.061+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:38.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:38.062+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:38.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:38.576+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:38.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:38.576+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:38.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:38.576+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:38.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:38.576+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:38.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:38.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:38.878+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:38.879+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:39.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:39.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:39.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:39.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:39.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:39.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:39.084+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:39.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:39.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:39.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:39.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:39.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:39.531+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:39.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:39.532+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:40.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:40.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:40.087+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:40.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:40.087+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:40.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:39.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:40.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:39.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:40.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:39.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:40.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:39.879+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:40.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:40.527+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:40.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:40.527+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:40.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:40.527+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:40.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:40.528+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:40.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:40.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:40.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:40.903+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:41.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:41.045+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:41.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:41.045+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:41.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:41.045+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:41.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:41.046+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:41.886 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:23:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:23:41.783+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 37495 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:23:41.886 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:41.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:41.887 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:41.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:41.887 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:41.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:41.887 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:41.564+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:42.007+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:42.007+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:42.007+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:42.008+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:41.883+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:42.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:41.883+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:42.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:41.883+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:42.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:41.884+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:42.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:42.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:42.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:42.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:42.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:42.609+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:42.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:42.610+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:42.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:42.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:42.880+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:42.881+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:43.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:43.041+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:43.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:43.041+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:43.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:43.041+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:43.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:43.041+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:43.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:43.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:43.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:43.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:43.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:43.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:43.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:43.578+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:44.045+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:44.045+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:44.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:44.045+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:44.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:44.046+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:44.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:43.914+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:44.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:43.914+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:44.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:43.914+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:44.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:43.915+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:44.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:44.551+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:44.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:44.551+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:44.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:44.551+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:44.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:44.552+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:45.094+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:45.094+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:45.094+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:45.095+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:45.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:44.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:45.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:44.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:45.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:44.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:45.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:44.958+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:45.511+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:45.511+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:45.511+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:45.511+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:45.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:45.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:45.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:45.988+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:46.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:46.063+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:46.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:46.063+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:46.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:46.063+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:46.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:46.063+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:46.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:46.497+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:46.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:46.497+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:46.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:46.497+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:46.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:46.498+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:47.107 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:23:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:23:46.788+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 37586 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:23:47.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:47.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:47.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:47.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:47.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:47.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:47.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:47.023+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:47.104+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:47.104+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:47.104+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:47.105+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:47.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:47.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:47.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:47.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:47.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:47.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:47.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:47.489+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:48.442 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:47.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:48.443 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:47.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:48.443 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:47.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:48.443 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:47.994+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:48.443 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:48.135+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:48.443 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:48.135+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:48.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:48.135+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:48.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:48.135+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:48.440+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:48.440+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:48.440+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:48.440+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:49.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:49.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:49.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:49.141+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:48.956+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:49.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:48.956+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:49.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:48.956+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:49.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:48.957+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:49.417+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:49.417+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:49.417+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:49.418+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:50.386 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:49.996+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:50.387 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:49.996+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:50.387 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:49.996+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:50.387 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:49.997+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:50.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:50.154+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:50.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:50.154+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:50.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:50.154+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:50.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:50.155+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:50.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:50.383+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:50.383+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:50.383+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:50.384+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:51.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:50.976+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:51.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:50.976+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:51.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:50.976+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:51.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:50.977+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:51.426 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:51.137+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:51.426 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:51.137+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:51.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:51.137+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:51.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:51.138+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:51.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:51.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:51.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:51.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:51.697 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:51.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:51.697 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:51.425+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:51.947 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:23:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:23:51.793+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 37676 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:23:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:51.944+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:51.944+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:51.944+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:51.945+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:52.147+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:52.147+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:52.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:52.147+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:52.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:52.148+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:52.425+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:52.425+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:52.425+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:52.426+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:53.132+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:53.132+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:53.132+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:53.133+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:53.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:52.898+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:53.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:52.898+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:53.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:52.898+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:53.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:52.899+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:53.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:53.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:53.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:53.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:53.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:53.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:53.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:53.438+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:54.117+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:54.117+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:54.117+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:54.118+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:54.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:53.899+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:54.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:53.899+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:54.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:53.899+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:54.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:53.899+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:54.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:54.472+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:54.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:54.472+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:54.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:54.472+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:54.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:54.473+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:55.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:54.885+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:55.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:54.885+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:55.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:54.885+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:55.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:54.886+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:55.071+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:55.071+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:55.071+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:55.072+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:55.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:55.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:55.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:55.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:55.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:55.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:55.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:55.507+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:55.925+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:55.925+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:55.925+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:55.926+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:56.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:56.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:56.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:56.051+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:56.800 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:56.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:56.800 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:56.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:56.800 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:56.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:56.801 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:56.469+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:57.091 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:23:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:23:56.797+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 37766 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:23:57.091 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:56.942+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:57.091 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:56.942+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:57.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:56.942+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:57.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:56.943+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:57.462 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:57.088+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:57.463 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:57.088+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:57.463 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:57.088+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:57.463 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:57.089+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:57.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:57.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:57.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:57.461+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:58.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:58.138+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:58.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:58.138+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:58.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:58.138+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:58.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:58.139+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:58.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:57.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:58.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:57.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:58.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:57.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:58.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:57.971+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:58.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:58.467+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:58.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:58.467+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:58.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:58.467+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:58.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:58.468+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:59.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:59.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:59.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:23:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:23:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:23:59.136+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:23:59.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:58.959+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:59.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:58.959+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:59.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:58.959+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:23:59.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:58.959+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:23:59.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:59.487+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:59.952 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:59.487+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:59.952 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:59.487+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:23:59.952 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:23:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:23:59.488+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:00.163+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:00.163+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:00.163+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:00.163+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:00.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:59.949+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:00.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:59.949+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:00.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:59.949+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:00.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:23:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:23:59.950+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:00.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:00.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:00.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:00.480+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:01.214 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:00.901+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:01.214 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:00.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:01.214 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:00.902+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:01.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:00.902+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:01.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:01.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:01.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:01.431+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:01.211+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:01.211+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:01.211+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:01.212+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:02.192 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:24:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:24:01.803+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 37857 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:24:02.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:01.952+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:02.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:01.952+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:02.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:01.952+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:02.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:01.953+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:02.190+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:02.190+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:02.190+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:02.191+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:02.386+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:02.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:02.386+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:02.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:02.386+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:02.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:02.386+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:03.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:02.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:03.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:02.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:03.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:02.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:03.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:02.973+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:03.280 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:03.204+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:03.280 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:03.204+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:03.280 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:03.204+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:03.280 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:03.205+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:03.662 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:03.401+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:03.662 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:03.401+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:03.662 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:03.401+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:03.662 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:03.402+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:03.942+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:03.942+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:03.942+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:03.943+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:04.452+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:04.452+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:04.452+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:04.452+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:04.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:04.233+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:04.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:04.233+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:04.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:04.233+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:04.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:04.234+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:05.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:04.978+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:05.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:04.978+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:05.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:04.978+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:05.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:04.978+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:05.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:05.213+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:05.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:05.213+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:05.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:05.213+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:05.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:05.214+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:05.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:05.435+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:05.435+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:05.435+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:05.436+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:06.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:05.974+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:06.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:05.974+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:06.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:05.974+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:06.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:05.974+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:06.432+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:06.432+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:06.432+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:06.432+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:06.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:06.233+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:06.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:06.233+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:06.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:06.233+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:06.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:06.234+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:07.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:24:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:24:06.807+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 37948 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:24:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:06.963+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:06.963+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:06.963+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:06.963+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:07.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:07.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:07.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:07.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:07.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:07.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:07.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:07.232+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:07.400+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:07.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:07.400+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:07.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:07.400+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:07.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:07.400+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:08.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:08.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:08.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:08.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:08.278 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:08.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:08.278 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:08.003+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:08.362+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:08.362+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:08.362+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:08.363+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:08.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:08.275+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:08.275+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:08.275+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:08.275+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:09.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:08.982+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:09.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:08.982+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:09.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:08.982+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:09.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:08.983+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:09.351+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:09.351+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:09.351+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:09.352+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:09.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:09.249+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:09.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:09.249+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:09.249+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:09.250+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:10.247 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:09.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:10.247 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:09.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:10.247 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:09.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:10.248 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:09.994+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:10.337+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:10.337+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:10.337+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:10.338+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:10.259+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:10.259+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:10.259+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:10.260+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:11.311 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:11.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:11.311 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:11.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:11.311 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:11.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:11.312 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:11.003+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:11.388+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:11.388+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:11.388+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:11.388+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:11.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:11.308+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:11.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:11.308+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:11.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:11.308+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:11.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:11.308+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:12.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:24:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:24:11.811+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 38038 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:24:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:12.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:12.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:12.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:12.001+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:12.410+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:12.410+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:12.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:12.410+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:12.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:12.411+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:12.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:12.310+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:12.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:12.310+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:12.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:12.310+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:12.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:12.311+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:12.952+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:12.952+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:12.952+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:13.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:12.952+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:13.662 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:13.444+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:13.662 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:13.444+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:13.662 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:13.444+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:13.662 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:13.445+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:13.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:13.304+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:13.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:13.304+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:13.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:13.304+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:13.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:13.304+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:13.959+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:13.959+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:13.959+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:14.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:13.959+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:14.426+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:14.426+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:14.426+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:14.427+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:14.295+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:14.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:14.295+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:14.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:14.295+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:14.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:14.295+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:14.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:14.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:14.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:14.916+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:15.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:15.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:15.415+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:15.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:15.415+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:15.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:15.304+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:15.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:15.305+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:15.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:15.305+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:15.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:15.305+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:15.884+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:15.884+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:15.884+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:15.885+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:16.373+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:16.373+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:16.373+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:16.374+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:16.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:16.284+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:16.284+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:16.284+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:16.285+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:17.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:24:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:24:16.817+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 38129 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:24:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:16.923+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:16.923+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:16.923+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:16.924+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:17.373+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:17.373+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:17.373+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:17.374+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:17.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:17.283+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:17.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:17.283+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:17.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:17.283+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:17.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:17.284+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:17.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:17.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:17.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:17.931+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:18.563 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:18.414+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:18.563 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:18.414+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:18.563 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:18.414+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:18.563 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:18.415+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:18.564 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:18.241+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:18.564 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:18.241+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:18.564 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:18.241+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:18.564 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:18.241+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:18.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:18.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:18.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:18.963+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:19.249+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:19.249+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:19.249+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:19.249+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:19.372+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:19.372+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:19.372+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:19.372+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:19.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:19.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:19.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:19.963+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:20.391+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:20.391+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:20.391+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:20.392+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:20.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:20.293+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:20.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:20.293+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:20.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:20.293+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:20.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:20.294+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:20.924+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:20.924+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:20.924+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:20.924+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:21.349+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:21.349+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:21.349+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:21.349+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:21.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:21.252+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:21.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:21.252+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:21.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:21.252+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:21.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:21.252+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:22.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:24:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:24:21.822+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 38219 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:24:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:21.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:21.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:21.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:21.938+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:22.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:22.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:22.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:22.321+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:22.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:22.219+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:22.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:22.219+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:22.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:22.219+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:22.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:22.219+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:23.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:22.928+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:23.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:22.928+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:23.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:22.928+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:23.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:22.929+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:23.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:23.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:23.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:23.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:23.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:23.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:23.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:23.171+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:23.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:23.360+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:23.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:23.360+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:23.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:23.360+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:23.638 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:23.360+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:24.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:23.953+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:24.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:23.953+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:24.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:23.953+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:24.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:23.954+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:24.133+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:24.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:24.133+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:24.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:24.133+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:24.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:24.134+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:24.369+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:24.369+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:24.369+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:24.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:24.369+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:25.370 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:25.149+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:25.370 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:25.149+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:25.370 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:25.149+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:25.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:25.150+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:25.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:25.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:25.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:25.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:25.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:25.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:25.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:25.001+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:25.368+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:25.368+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:25.368+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:25.368+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:26.420 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:26.135+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:26.420 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:26.135+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:26.420 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:26.135+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:26.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:26.136+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:26.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:25.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:26.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:25.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:26.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:25.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:26.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:25.971+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:26.417+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:26.417+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:26.417+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:26.418+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:27.111 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:24:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:24:26.827+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 38310 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:24:27.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:26.975+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:27.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:26.975+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:27.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:26.975+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:27.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:26.976+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:27.375 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:27.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:27.375 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:27.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:27.375 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:27.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:27.375 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:27.110+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:27.372+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:27.372+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:27.372+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:27.373+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:27.945+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:27.945+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:27.945+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:27.946+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:28.128+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:28.128+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:28.128+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:28.129+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:28.712 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:28.378+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:28.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:28.378+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:28.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:28.378+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:28.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:28.379+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:29.389 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:28.989+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:29.390 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:28.989+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:29.390 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:28.989+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:29.390 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:28.990+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:29.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:29.102+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:29.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:29.102+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:29.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:29.102+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:29.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:29.103+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:29.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:29.387+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:29.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:29.387+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:29.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:29.387+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:29.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:29.388+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:29.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:29.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:29.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:29.959+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:30.118+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:30.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:30.118+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:30.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:30.118+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:30.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:30.119+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:30.344+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:30.344+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:30.344+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:30.345+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:31.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:30.963+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:31.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:30.963+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:31.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:30.963+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:31.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:30.963+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:31.309+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:31.309+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:31.309+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:31.310+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:31.111+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:31.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:31.111+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:31.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:31.111+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:31.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:31.112+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:31.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:24:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:24:31.833+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 38400 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:24:32.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:32.126+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:32.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:32.126+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:32.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:32.126+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:32.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:32.127+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:32.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:31.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:32.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:31.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:32.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:31.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:32.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:31.998+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:32.286+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:32.286+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:32.286+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:32.287+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:33.278 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:33.004+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:33.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:33.004+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:33.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:33.004+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:33.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:33.004+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:33.279 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:33.098+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:33.279 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:33.098+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:33.279 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:33.098+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:33.280 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:33.099+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:33.663 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:33.276+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:33.664 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:33.276+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:33.664 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:33.276+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:33.664 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:33.276+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:34.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:34.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:34.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:34.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:34.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:34.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:34.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:34.023+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:34.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:34.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:34.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:34.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:34.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:34.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:34.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:34.142+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:34.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:34.277+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:34.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:34.277+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:34.705 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:34.277+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:34.705 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:34.278+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:35.292+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:35.292+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:35.292+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:35.293+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:35.048+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:35.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:35.048+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:35.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:35.048+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:35.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:35.048+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:35.133+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:35.133+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:35.133+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:35.134+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:36.271+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:36.271+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:36.271+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:36.271+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:36.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:36.094+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:36.094+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:36.094+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:36.094+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:36.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:36.024+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:36.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:36.024+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:36.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:36.024+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:36.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:36.025+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:37.125 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:24:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:24:36.838+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 38490 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:24:37.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:37.034+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:37.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:37.034+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:37.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:37.034+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:37.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:37.034+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:37.122+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:37.122+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:37.122+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:37.123+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:37.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:37.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:37.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:37.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:37.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:37.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:37.273+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:38.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:38.248+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:38.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:38.248+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:38.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:38.248+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:38.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:38.248+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:38.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:38.031+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:38.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:38.031+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:38.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:38.031+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:38.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:38.032+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:38.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:38.144+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:38.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:38.144+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:38.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:38.144+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:38.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:38.144+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:39.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:39.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:39.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:39.068+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:39.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:39.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:39.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:39.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:39.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:39.131+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:39.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:39.131+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:39.218+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:39.219+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:39.219+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:39.219+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:40.069+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:40.069+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:40.069+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:40.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:40.070+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:40.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:40.176+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:40.176+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:40.176+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:40.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:40.167+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:40.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:40.167+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:40.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:40.167+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:40.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:40.167+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:41.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:41.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:41.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:41.022+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:41.161+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:41.161+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:41.161+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:41.162+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:41.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:41.160+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:41.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:41.160+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:41.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:41.160+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:41.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:41.161+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:42.122 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:24:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:24:41.843+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 38580 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:24:42.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:42.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:42.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:42.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:42.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:42.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:42.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:42.039+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:42.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:42.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:42.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:42.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:42.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:42.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:42.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:42.120+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:42.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:42.194+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:42.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:42.194+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:42.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:42.194+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:42.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:42.194+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:43.068+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:43.068+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:43.068+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:43.069+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:43.145+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:43.145+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:43.145+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:43.145+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:43.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:43.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:43.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:43.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:43.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:43.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:43.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:43.136+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:44.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:44.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:44.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:44.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:44.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:44.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:44.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:44.053+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:44.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:44.137+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:44.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:44.137+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:44.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:44.137+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:44.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:44.138+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:44.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:44.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:44.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:44.181+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:45.102+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:45.102+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:45.102+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:45.102+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:45.182+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:45.182+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:45.182+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:45.182+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:45.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:45.196+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:45.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:45.196+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:45.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:45.196+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:45.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:45.196+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:46.125+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:46.125+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:46.125+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:46.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:46.126+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:46.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:46.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:46.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:46.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:46.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:46.143+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:46.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:46.144+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:46.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:46.162+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:46.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:46.162+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:46.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:46.162+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:46.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:46.163+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:47.105 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:24:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:24:46.848+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 38671 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:24:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:47.103+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:47.103+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:47.103+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:47.104+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:47.164+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:47.164+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:47.164+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:47.165+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:47.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:47.118+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:47.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:47.118+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:47.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:47.118+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:47.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:47.118+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:48.158+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:48.158+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:48.158+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:48.159+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:48.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:48.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:48.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:48.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:48.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:48.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:48.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:48.148+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:48.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:48.074+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:48.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:48.074+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:48.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:48.074+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:48.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:48.075+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:49.133+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:49.133+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:49.133+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:49.134+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:49.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:49.100+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:49.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:49.100+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:49.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:49.100+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:49.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:49.100+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:49.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:49.146+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:49.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:49.146+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:49.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:49.146+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:49.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:49.146+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:50.122+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:50.122+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:50.122+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:50.123+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:50.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:50.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:50.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:50.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:50.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:50.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:50.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:50.106+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:50.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:50.142+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:50.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:50.142+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:50.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:50.142+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:50.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:50.143+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:51.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:51.159+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:51.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:51.159+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:51.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:51.159+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:51.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:51.160+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:51.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:51.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:51.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:51.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:51.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:51.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:51.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:51.107+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:51.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:51.101+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:51.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:51.101+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:51.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:51.101+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:51.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:51.101+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:52.149 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:24:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:24:51.853+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 38761 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:24:52.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:52.095+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:52.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:52.095+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:52.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:52.095+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:52.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:52.095+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:52.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:52.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:52.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:52.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:52.147+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:52.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:52.172+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:52.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:52.172+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:52.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:52.172+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:52.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:52.173+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:53.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:53.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:53.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:53.103+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:53.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:53.217+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:53.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:53.217+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:53.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:53.217+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:53.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:53.218+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:53.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:53.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:53.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:53.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:53.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:53.110+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:53.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:53.111+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:54.253+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:54.253+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:54.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:54.253+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:54.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:54.254+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:54.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:54.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:54.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:54.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:54.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:54.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:54.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:54.088+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:54.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:54.065+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:54.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:54.065+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:54.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:54.065+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:54.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:54.065+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:55.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:55.230+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:55.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:55.230+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:55.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:55.230+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:55.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:55.231+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:55.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:55.074+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:55.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:55.074+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:55.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:55.074+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:55.469 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:55.075+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:55.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:55.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:55.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:55.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:55.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:55.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:55.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:55.070+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:56.259+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:56.259+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:56.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:56.259+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:56.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:56.260+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:56.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:56.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:56.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:56.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:56.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:56.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:56.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:56.114+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:56.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:56.027+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:56.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:56.027+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:56.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:56.027+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:56.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:56.028+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:57.142 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:24:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:24:56.857+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 38850 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:24:57.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:57.045+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:57.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:57.045+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:57.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:57.045+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:57.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:57.046+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:57.138+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:57.138+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:57.138+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:57.139+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:57.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:57.268+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:57.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:57.268+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:57.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:57.268+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:57.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:57.269+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:58.246+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:58.246+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:58.246+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:58.246+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:58.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:58.004+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:58.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:58.004+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:58.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:58.004+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:58.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:58.004+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:58.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:58.116+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:58.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:58.116+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:58.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:58.116+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:58.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:58.117+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:59.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:59.264+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:59.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:59.264+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:59.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:59.264+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:24:59.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:24:59.265+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:24:59.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:59.034+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:59.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:59.034+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:59.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:59.034+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:24:59.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:59.035+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:24:59.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:59.145+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:59.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:59.145+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:59.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:59.145+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:24:59.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:24:59.146+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:00.282 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:59.997+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:00.282 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:59.997+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:00.282 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:59.997+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:00.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:24:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:24:59.998+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:00.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:00.179+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:00.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:00.179+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:00.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:00.179+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:00.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:00.180+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:00.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:00.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:00.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:00.280+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:01.280 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:01.010+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:01.280 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:01.010+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:01.280 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:01.010+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:01.280 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:01.011+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:01.280 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:01.150+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:01.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:01.150+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:01.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:01.150+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:01.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:01.151+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:01.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:01.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:01.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:01.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:01.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:01.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:01.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:01.278+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:02.169 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:25:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:25:01.863+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 38940 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:25:02.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:02.051+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:02.170 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:02.051+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:02.170 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:02.051+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:02.170 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:02.052+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:02.167+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:02.167+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:02.167+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:02.167+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:02.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:02.259+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:02.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:02.259+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:02.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:02.259+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:02.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:02.260+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:03.057+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:03.057+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:03.057+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:03.058+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:03.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:03.269+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:03.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:03.269+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:03.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:03.269+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:03.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:03.269+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:03.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:03.207+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:03.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:03.207+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:03.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:03.207+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:03.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:03.207+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:04.234+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:04.234+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:04.234+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:04.234+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:04.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:04.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:04.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:04.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:04.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:04.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:04.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:04.229+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:04.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:04.071+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:04.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:04.071+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:04.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:04.071+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:04.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:04.071+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:05.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:05.262+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:05.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:05.262+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:05.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:05.262+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:05.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:05.263+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:05.425 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:05.028+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:05.425 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:05.028+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:05.425 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:05.028+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:05.426 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:05.029+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:05.426 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:05.237+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:05.426 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:05.237+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:05.426 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:05.237+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:05.426 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:05.237+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:06.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:06.225+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:06.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:06.225+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:06.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:06.225+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:06.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:06.225+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:06.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:06.039+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:06.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:06.039+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:06.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:06.039+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:06.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:06.040+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:06.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:06.234+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:06.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:06.234+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:06.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:06.234+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:06.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:06.235+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:07.204 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:25:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:25:06.868+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 39031 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:25:07.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:07.065+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:07.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:07.065+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:07.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:07.065+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:07.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:07.065+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:07.201+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:07.201+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:07.201+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:07.202+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:07.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:07.261+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:07.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:07.261+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:07.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:07.261+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:07.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:07.261+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:08.112+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:08.112+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:08.112+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:08.113+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:08.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:08.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:08.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:08.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:08.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:08.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:08.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:08.278+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:08.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:08.172+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:08.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:08.172+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:08.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:08.172+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:08.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:08.173+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:09.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:09.064+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:09.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:09.064+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:09.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:09.064+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:09.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:09.064+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:09.387 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:09.250+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:09.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:09.250+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:09.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:09.250+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:09.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:09.251+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:09.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:09.142+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:09.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:09.142+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:09.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:09.142+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:09.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:09.142+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:10.053+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:10.053+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:10.053+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:10.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:10.054+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:10.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:10.250+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:10.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:10.250+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:10.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:10.250+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:10.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:10.251+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:10.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:10.114+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:10.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:10.114+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:10.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:10.114+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:10.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:10.114+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:11.267+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:11.267+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:11.267+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:11.268+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:11.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:11.107+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:11.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:11.107+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:11.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:11.107+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:11.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:11.107+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:11.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:11.057+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:11.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:11.057+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:11.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:11.057+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:11.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:11.058+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:12.138 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:25:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:25:11.873+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 39121 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:25:12.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:12.023+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:12.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:12.023+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:12.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:12.023+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:12.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:12.023+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:12.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:12.234+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:12.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:12.234+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:12.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:12.234+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:12.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:12.234+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:12.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:12.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:12.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:12.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:12.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:12.136+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:12.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:12.136+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:13.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:13.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:13.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:13.276+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:13.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:13.036+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:13.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:13.036+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:13.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:13.036+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:13.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:13.037+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:13.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:13.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:13.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:13.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:13.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:13.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:13.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:13.120+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:14.324 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:13.995+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:14.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:13.995+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:14.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:13.995+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:14.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:13.995+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:14.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:14.160+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:14.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:14.160+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:14.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:14.160+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:14.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:14.161+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:14.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:14.322+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:14.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:14.322+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:14.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:14.322+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:14.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:14.323+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:15.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:15.130+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:15.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:15.130+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:15.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:15.130+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:15.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:15.130+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:15.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:15.000+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:15.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:15.000+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:15.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:15.000+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:15.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:15.001+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:15.368+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:15.368+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:15.368+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:15.369+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:16.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:15.964+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:16.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:15.964+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:16.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:15.964+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:16.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:15.964+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:16.378 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:16.090+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:16.378 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:16.090+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:16.378 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:16.090+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:16.378 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:16.091+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:16.375+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:16.375+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:16.375+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:16.375+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:17.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:25:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:25:16.878+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 39210 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:25:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:16.981+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:16.981+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:16.981+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:17.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:16.982+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:17.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:17.126+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:17.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:17.126+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:17.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:17.126+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:17.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:17.127+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:17.476 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:17.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:17.476 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:17.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:17.476 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:17.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:17.476 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:17.388+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:18.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:17.989+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:18.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:17.989+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:18.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:17.990+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:18.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:17.990+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:18.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:18.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:18.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:18.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:18.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:18.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:18.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:18.082+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:18.354+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:18.354+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:18.354+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:18.354+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:19.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:19.025+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:19.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:19.025+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:19.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:19.025+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:19.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:19.026+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:19.381 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:19.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:19.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:19.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:19.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:19.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:19.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:19.119+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:19.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:19.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:19.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:19.379+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:20.251 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:19.988+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:20.251 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:19.988+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:20.251 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:19.988+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:20.251 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:19.989+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:20.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:20.135+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:20.252 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:20.135+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:20.252 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:20.135+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:20.252 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:20.135+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:20.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:20.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:20.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:20.378+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:21.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:20.943+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:21.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:20.943+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:21.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:20.943+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:21.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:20.943+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:21.157+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:21.157+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:21.157+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:21.158+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:21.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:21.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:21.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:21.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:21.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:21.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:21.330+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:22.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:25:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:25:21.883+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 39299 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:25:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:21.939+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:21.939+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:21.939+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:22.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:21.940+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:22.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:22.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:22.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:22.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:22.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:22.119+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:22.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:22.120+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:22.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:22.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:22.311+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:22.312+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:23.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:22.988+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:23.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:22.988+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:23.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:22.988+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:23.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:22.989+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:23.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:23.108+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:23.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:23.108+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:23.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:23.108+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:23.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:23.109+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:23.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:23.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:23.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:23.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:23.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:23.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:23.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:23.331+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:24.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:23.967+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:24.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:23.967+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:24.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:23.967+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:24.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:23.968+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:24.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:24.105+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:24.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:24.105+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:24.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:24.105+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:24.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:24.105+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:24.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:24.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:24.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:24.379+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:25.396 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:25.000+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:25.396 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:25.000+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:25.397 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:25.000+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:25.397 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:25.000+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:25.397 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:25.118+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:25.397 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:25.118+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:25.397 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:25.118+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:25.397 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:25.119+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:25.394+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:25.394+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:25.394+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:25.394+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:26.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:26.030+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:26.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:26.030+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:26.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:26.030+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:26.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:26.031+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:26.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:26.128+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:26.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:26.129+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:26.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:26.129+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:26.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:26.129+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:26.399+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:26.399+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:26.399+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:26.400+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:27.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:25:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:25:26.888+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 39390 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:25:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:27.122+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:27.122+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:27.122+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:27.123+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:27.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:27.065+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:27.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:27.065+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:27.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:27.065+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:27.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:27.065+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:27.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:27.430+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:27.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:27.430+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:27.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:27.430+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:27.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:27.431+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:28.412 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:28.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:28.413 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:28.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:28.413 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:28.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:28.413 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:28.060+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:28.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:28.156+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:28.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:28.156+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:28.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:28.156+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:28.414 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:28.157+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:28.410+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:28.410+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:28.410+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:28.411+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:29.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:29.079+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:29.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:29.079+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:29.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:29.079+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:29.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:29.080+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:29.425 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:29.174+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:29.425 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:29.174+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:29.425 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:29.174+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:29.425 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:29.175+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:29.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:29.421+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:29.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:29.421+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:29.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:29.421+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:29.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:29.422+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:30.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:30.182+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:30.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:30.182+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:30.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:30.182+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:30.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:30.183+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:30.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:30.082+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:30.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:30.082+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:30.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:30.082+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:30.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:30.083+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:30.437+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:30.437+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:30.437+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:30.437+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:31.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:31.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:31.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:31.105+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:31.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:31.177+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:31.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:31.177+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:31.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:31.177+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:31.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:31.178+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:31.896 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:31.479+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:31.896 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:31.479+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:31.896 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:31.479+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:31.896 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:31.480+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:32.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:25:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:25:31.893+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 39478 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:25:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:32.133+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:32.133+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:32.133+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:32.133+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:32.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:32.217+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:32.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:32.217+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:32.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:32.217+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:32.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:32.217+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:32.838 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:32.528+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:32.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:32.528+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:32.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:32.528+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:32.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:32.529+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:33.092+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:33.092+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:33.092+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:33.093+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:33.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:33.259+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:33.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:33.259+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:33.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:33.259+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:33.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:33.260+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:33.811 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:33.489+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:33.811 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:33.489+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:33.811 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:33.489+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:33.811 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:33.490+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:34.123+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:34.123+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:34.123+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:34.124+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:34.300+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:34.300+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:34.300+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:34.301+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:34.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:34.490+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:34.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:34.490+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:34.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:34.490+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:34.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:34.490+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:35.082+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:35.082+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:35.082+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:35.083+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:35.309+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:35.309+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:35.309+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:35.309+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:35.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:35.469+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:35.469+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:35.469+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:35.470+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:36.079+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:36.079+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:36.079+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:36.079+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:36.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:36.286+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:36.286+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:36.286+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:36.287+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:36.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:36.499+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:36.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:36.499+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:36.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:36.499+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:36.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:36.499+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:37.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:25:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:25:36.899+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 39568 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:25:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:37.057+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:37.057+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:37.057+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:37.058+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:37.523+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:37.523+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:37.523+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:37.523+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:37.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:37.323+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:37.323+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:37.323+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:37.323+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:38.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:38.060+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:38.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:38.060+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:38.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:38.060+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:38.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:38.061+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:38.506+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:38.506+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:38.506+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:38.507+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:38.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:38.332+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:38.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:38.332+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:38.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:38.332+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:38.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:38.333+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:39.343 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:39.045+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:39.343 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:39.045+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:39.343 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:39.045+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:39.343 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:39.045+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:39.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:39.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:39.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:39.474+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:39.340+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:39.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:39.340+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:39.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:39.340+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:39.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:39.341+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:40.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:40.001+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:40.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:40.001+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:40.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:40.001+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:40.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:40.001+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:40.438+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:40.438+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:40.438+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:40.439+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:40.315+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:40.315+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:40.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:40.315+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:40.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:40.315+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:41.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:40.965+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:41.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:40.965+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:41.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:40.965+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:41.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:40.965+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:41.399+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:41.399+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:41.399+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:41.400+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:41.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:41.338+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:41.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:41.338+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:41.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:41.338+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:41.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:41.339+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:42.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:25:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:25:41.903+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 39658 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:25:42.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:41.933+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:42.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:41.933+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:42.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:41.933+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:42.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:41.934+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:42.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:42.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:42.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:42.363+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:42.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:42.386+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:42.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:42.386+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:42.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:42.386+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:42.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:42.387+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:43.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:42.952+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:43.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:42.952+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:43.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:42.952+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:43.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:42.953+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:43.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:43.399+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:43.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:43.399+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:43.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:43.399+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:43.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:43.400+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:43.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:43.412+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:43.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:43.412+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:43.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:43.412+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:43.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:43.412+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:44.329 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:43.977+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:44.329 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:43.977+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:44.329 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:43.977+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:44.330 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:43.977+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:44.382+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:44.382+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:44.382+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:44.383+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:44.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:44.393+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:44.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:44.393+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:44.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:44.393+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:44.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:44.394+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:45.012+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:45.012+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:45.012+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:45.012+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:45.337+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:45.337+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:45.337+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:45.337+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:45.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:45.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:45.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:45.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:45.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:45.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:45.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:45.387+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:46.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:45.972+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:46.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:45.972+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:46.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:45.972+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:46.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:45.973+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:46.423+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:46.423+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:46.423+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:46.423+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:46.386+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:46.386+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:46.386+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:46.387+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:47.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:25:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:25:46.909+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 39748 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:25:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:46.987+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:46.987+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:46.987+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:46.988+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:47.478 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:47.354+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:47.478 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:47.354+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:47.478 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:47.354+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:47.478 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:47.355+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:47.478 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:47.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:47.479 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:47.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:47.479 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:47.418+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:47.479 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:47.418+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:48.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:48.015+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:48.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:48.015+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:48.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:48.015+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:48.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:48.016+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:48.377+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:48.377+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:48.377+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:48.377+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:48.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:48.324+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:48.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:48.324+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:48.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:48.324+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:48.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:48.324+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:49.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:49.051+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:49.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:49.051+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:49.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:49.051+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:49.406 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:49.052+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:49.406 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:49.295+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:49.406 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:49.295+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:49.406 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:49.295+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:49.406 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:49.295+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:49.403+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:49.403+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:49.403+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:49.403+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:50.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:50.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:50.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:50.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:50.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:50.102+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:50.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:50.103+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:50.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:50.426+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:50.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:50.426+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:50.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:50.426+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:50.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:50.427+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:50.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:50.282+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:50.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:50.282+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:50.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:50.282+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:50.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:50.283+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:51.435 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:51.068+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:51.435 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:51.068+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:51.436 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:51.068+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:51.436 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:51.069+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:51.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:51.263+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:51.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:51.263+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:51.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:51.263+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:51.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:51.264+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:51.433+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:51.433+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:51.433+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:51.433+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:52.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:25:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:25:51.914+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 39839 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:25:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:52.051+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:52.051+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:52.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:52.051+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:52.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:52.051+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:52.474+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:52.474+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:52.474+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:52.475+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:52.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:52.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:52.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:52.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:52.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:52.233+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:53.073+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:53.073+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:53.073+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:53.074+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:53.227+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:53.227+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:53.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:53.227+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:53.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:53.227+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:53.478+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:53.478+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:53.478+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:53.478+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:54.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:54.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:54.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:54.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:54.181+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:54.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:54.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:54.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:54.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:54.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:54.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:54.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:54.089+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:54.519+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:54.519+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:54.519+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:54.520+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:55.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:55.098+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:55.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:55.098+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:55.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:55.098+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:55.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:55.099+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:55.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:55.177+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:55.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:55.177+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:55.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:55.177+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:55.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:55.178+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:55.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:55.488+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:55.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:55.488+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:55.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:55.488+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:55.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:55.489+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:56.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:56.104+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:56.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:56.104+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:56.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:56.104+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:56.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:56.104+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:56.191+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:56.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:56.191+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:56.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:56.191+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:56.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:56.191+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:56.922 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:56.472+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:56.922 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:56.472+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:56.922 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:56.472+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:56.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:56.473+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:57.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:25:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:25:56.919+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 39929 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:25:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:57.133+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:57.133+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:57.133+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:57.134+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:57.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:57.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:57.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:57.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:57.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:57.232+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:57.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:57.232+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:57.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:57.511+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:57.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:57.511+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:57.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:57.511+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:57.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:57.512+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:58.085+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:58.085+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:58.085+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:58.086+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:58.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:58.223+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:58.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:58.223+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:58.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:58.223+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:58.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:58.223+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:58.545+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:58.545+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:58.545+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:58.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:58.545+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:25:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:59.067+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:59.067+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:59.067+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:25:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:25:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:25:59.068+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:59.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:59.244+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:59.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:59.244+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:59.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:59.244+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:25:59.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:25:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:25:59.245+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:25:59.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:59.504+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:59.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:59.504+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:59.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:59.504+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:25:59.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:25:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:25:59.505+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:00.022+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:00.022+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:00.022+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:00.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:00.023+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:00.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:00.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:00.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:00.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:00.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:00.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:00.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:00.230+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:00.529+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:00.529+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:00.529+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:00.530+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:01.012+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:01.012+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:01.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:01.012+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:01.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:01.012+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:01.254+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:01.254+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:01.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:01.254+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:01.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:01.255+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:01.927 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:01.482+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:01.927 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:01.482+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:01.928 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:01.482+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:01.928 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:01.482+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:02.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:26:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:26:01.925+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 40019 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:26:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:02.015+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:02.015+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:02.015+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:02.015+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:02.457+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:02.457+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:02.457+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:02.458+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:02.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:02.293+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:02.293+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:02.293+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:02.294+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:03.330 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:03.025+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:03.330 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:03.025+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:03.330 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:03.025+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:03.331 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:03.026+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:03.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:03.501+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:03.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:03.501+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:03.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:03.501+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:03.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:03.501+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:03.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:03.328+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:03.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:03.328+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:03.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:03.328+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:03.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:03.328+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:04.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:04.004+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:04.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:04.004+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:04.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:04.004+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:04.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:04.005+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:04.523+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:04.523+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:04.523+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:04.524+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:04.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:04.369+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:04.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:04.369+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:04.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:04.369+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:04.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:04.369+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:05.299 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:05.004+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:05.300 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:05.004+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:05.300 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:05.004+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:05.300 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:05.005+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:05.516+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:05.516+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:05.516+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:05.517+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:05.404+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:05.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:05.404+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:05.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:05.404+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:05.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:05.405+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:06.412 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:06.033+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:06.413 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:06.033+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:06.413 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:06.033+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:06.413 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:06.034+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:06.515+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:06.515+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:06.515+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:06.516+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:06.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:06.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:06.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:06.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:06.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:06.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:06.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:06.411+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:07.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:26:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:26:06.930+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 40109 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:26:07.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:07.040+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:07.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:07.041+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:07.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:07.041+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:07.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:07.041+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:07.489+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:07.489+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:07.489+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:07.489+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:07.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:07.405+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:07.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:07.405+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:07.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:07.405+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:07.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:07.406+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:08.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:08.051+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:08.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:08.051+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:08.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:08.051+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:08.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:08.052+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:08.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:08.405+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:08.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:08.405+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:08.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:08.405+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:08.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:08.406+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:08.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:08.519+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:08.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:08.519+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:08.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:08.519+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:08.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:08.519+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:09.415 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:09.073+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:09.416 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:09.073+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:09.416 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:09.073+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:09.416 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:09.074+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:09.561+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:09.561+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:09.561+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:09.562+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:09.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:09.413+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:09.413+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:09.413+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:09.414+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:10.376 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:10.031+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:10.376 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:10.031+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:10.376 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:10.031+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:10.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:10.031+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:10.554+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:10.554+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:10.554+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:10.555+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:10.431+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:10.431+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:10.431+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:10.432+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:11.398 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:11.041+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:11.398 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:11.041+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:11.398 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:11.041+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:11.398 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:11.042+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:11.526+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:11.526+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:11.526+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:11.527+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:11.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:11.395+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:11.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:11.395+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:11.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:11.395+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:11.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:11.396+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:12.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:26:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:26:11.935+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 40199 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:26:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:12.065+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:12.065+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:12.065+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:12.065+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:12.355+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:12.355+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:12.355+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:12.356+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:12.493+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:12.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:12.493+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:12.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:12.493+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:12.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:12.494+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:13.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:13.115+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:13.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:13.115+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:13.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:13.115+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:13.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:13.116+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:13.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:13.468+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:13.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:13.468+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:13.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:13.468+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:13.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:13.468+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:13.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:13.332+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:13.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:13.332+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:13.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:13.332+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:13.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:13.333+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:14.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:14.324+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:14.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:14.324+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:14.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:14.324+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:14.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:14.325+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:14.458 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:14.161+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:14.458 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:14.161+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:14.458 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:14.161+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:14.458 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:14.161+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:14.454+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:14.454+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:14.454+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:14.455+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:15.431 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:15.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:15.431 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:15.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:15.431 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:15.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:15.431 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:15.175+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:15.431 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:15.280+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:15.432 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:15.280+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:15.432 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:15.280+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:15.432 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:15.281+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:15.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:15.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:15.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:15.429+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:16.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:16.263+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:16.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:16.263+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:16.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:16.263+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:16.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:16.264+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:16.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:16.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:16.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:16.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:16.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:16.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:16.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:16.215+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:16.398+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:16.398+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:16.398+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:16.398+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:17.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:26:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:26:16.940+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 40290 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:26:17.478 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:17.417+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:17.478 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:17.417+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:17.478 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:17.417+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:17.478 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:17.418+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:17.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:17.241+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:17.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:17.241+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:17.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:17.241+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:17.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:17.242+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:17.479 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:17.295+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:17.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:17.295+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:17.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:17.295+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:17.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:17.296+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:18.242+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:18.242+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:18.242+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:18.242+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:18.296+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:18.296+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:18.296+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:18.297+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:18.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:18.441+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:18.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:18.441+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:18.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:18.441+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:18.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:18.442+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:19.422+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:19.422+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:19.422+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:19.423+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:19.282+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:19.282+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:19.282+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:19.282+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:19.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:19.294+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:19.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:19.294+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:19.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:19.294+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:19.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:19.295+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:20.551 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:20.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:20.551 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:20.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:20.551 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:20.413+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:20.551 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:20.414+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:20.552 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:20.254+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:20.552 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:20.254+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:20.552 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:20.254+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:20.552 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:20.255+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:20.552 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:20.262+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:20.552 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:20.262+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:20.553 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:20.262+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:20.553 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:20.263+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:21.452+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:21.452+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:21.452+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:21.453+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:21.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:21.271+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:21.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:21.271+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:21.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:21.271+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:21.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:21.272+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:21.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:21.310+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:21.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:21.310+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:21.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:21.310+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:21.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:21.310+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:22.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:26:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:26:21.946+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 40379 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:26:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:22.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:22.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:22.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:22.273+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:22.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:22.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:22.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:22.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:22.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:22.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:22.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:22.412+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:22.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:22.328+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:22.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:22.328+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:22.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:22.328+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:22.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:22.329+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:23.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:23.318+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:23.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:23.318+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:23.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:23.318+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:23.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:23.319+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:23.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:23.457+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:23.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:23.457+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:23.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:23.457+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:23.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:23.458+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:23.671 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:23.297+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:23.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:23.297+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:23.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:23.297+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:23.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:23.298+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:24.436+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:24.436+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:24.436+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:24.437+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:24.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:24.312+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:24.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:24.312+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:24.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:24.312+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:24.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:24.312+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:24.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:24.340+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:24.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:24.340+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:24.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:24.340+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:24.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:24.341+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:25.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:25.296+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:25.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:25.296+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:25.626 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:25.296+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:25.626 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:25.297+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:25.626 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:25.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:25.626 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:25.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:25.626 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:25.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:25.626 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:25.471+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:25.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:25.338+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:25.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:25.338+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:25.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:25.338+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:25.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:25.339+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:26.515+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:26.515+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:26.515+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:26.516+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:26.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:26.337+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:26.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:26.337+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:26.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:26.337+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:26.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:26.338+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:26.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:26.329+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:26.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:26.329+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:26.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:26.329+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:26.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:26.330+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:27.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:26:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:26:26.950+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 40471 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:26:27.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:27.350+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:27.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:27.350+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:27.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:27.350+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:27.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:27.351+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:27.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:27.509+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:27.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:27.509+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:27.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:27.509+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:27.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:27.509+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:27.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:27.334+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:27.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:27.334+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:27.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:27.334+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:27.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:27.335+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:28.319+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:28.319+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:28.319+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:28.320+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:28.324+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:28.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:28.324+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:28.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:28.324+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:28.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:28.325+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:28.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:28.502+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:28.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:28.502+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:28.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:28.502+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:28.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:28.502+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:29.301+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:29.301+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:29.301+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:29.302+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:29.545+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:29.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:29.545+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:29.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:29.545+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:29.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:29.545+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:29.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:29.343+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:29.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:29.343+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:29.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:29.343+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:29.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:29.343+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:30.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:30.343+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:30.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:30.343+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:30.583 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:30.343+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:30.583 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:30.344+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:30.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:30.378+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:30.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:30.378+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:30.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:30.378+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:30.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:30.378+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:30.580+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:30.580+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:30.580+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:30.580+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:31.617 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:31.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:31.618 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:31.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:31.618 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:31.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:31.618 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:31.365+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:31.618 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:31.343+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:31.618 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:31.343+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:31.618 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:31.343+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:31.619 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:31.344+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:31.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:31.614+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:31.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:31.614+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:31.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:31.614+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:31.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:31.615+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:32.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:26:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:26:31.957+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 40560 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:26:32.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:32.296+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:32.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:32.296+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:32.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:32.296+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:32.612 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:32.297+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:32.612 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:32.348+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:32.612 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:32.348+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:32.612 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:32.348+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:32.612 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:32.349+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:32.609+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:32.609+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:32.609+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:32.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:32.609+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:33.592 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:33.305+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:33.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:33.305+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:33.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:33.305+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:33.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:33.305+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:33.594 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:33.339+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:33.594 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:33.339+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:33.594 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:33.339+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:33.594 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:33.340+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:33.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:33.590+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:33.590+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:33.590+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:33.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:33.590+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:34.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:34.269+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:34.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:34.269+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:34.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:34.269+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:34.611 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:34.270+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:34.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:34.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:34.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:34.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:34.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:34.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:34.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:34.337+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:34.607+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:34.607+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:34.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:34.607+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:34.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:34.608+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:35.594 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:35.223+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:35.595 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:35.223+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:35.595 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:35.223+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:35.595 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:35.223+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:35.595 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:35.351+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:35.595 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:35.351+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:35.595 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:35.351+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:35.596 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:35.352+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:35.592+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:35.592+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:35.592+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:35.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:35.593+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:36.205+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:36.205+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:36.205+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:36.206+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:36.349+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:36.349+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:36.349+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:36.350+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:36.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:36.581+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:36.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:36.581+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:36.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:36.581+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:36.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:36.582+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:37.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:26:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:26:36.962+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 40651 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:26:37.479 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:37.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:37.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:37.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:37.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:37.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:37.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:37.359+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:37.480 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:37.215+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:37.480 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:37.215+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:37.481 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:37.215+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:37.481 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:37.215+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:37.606+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:37.606+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:37.606+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:37.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:37.606+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:38.626 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:38.247+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:38.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:38.247+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:38.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:38.247+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:38.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:38.248+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:38.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:38.344+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:38.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:38.344+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:38.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:38.344+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:38.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:38.345+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:38.624+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:38.624+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:38.624+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:38.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:38.624+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:39.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:39.385+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:39.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:39.385+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:39.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:39.385+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:39.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:39.386+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:39.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:39.207+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:39.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:39.207+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:39.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:39.207+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:39.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:39.208+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:39.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:39.643+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:39.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:39.643+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:39.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:39.643+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:39.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:39.644+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:40.358+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:40.358+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:40.358+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:40.358+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:40.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:40.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:40.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:40.174+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:40.681+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:40.681+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:40.681+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:40.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:40.682+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:41.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:41.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:41.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:41.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:41.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:41.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:41.159+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:41.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:41.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:41.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:41.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:41.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:41.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:41.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:41.359+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:41.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:41.661+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:41.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:41.661+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:41.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:41.661+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:41.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:41.662+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:42.261 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:26:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:26:41.968+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 40739 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:26:42.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:42.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:42.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:42.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:42.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:42.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:42.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:42.154+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:42.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:42.360+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:42.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:42.360+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:42.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:42.360+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:42.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:42.361+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:42.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:42.702+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:42.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:42.702+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:42.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:42.702+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:42.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:42.703+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:43.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:43.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:43.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:43.155+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:43.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:43.346+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:43.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:43.346+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:43.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:43.346+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:43.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:43.347+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:43.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:43.704+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:43.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:43.704+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:43.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:43.704+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:43.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:43.705+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:44.189+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:44.189+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:44.189+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:44.189+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:44.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:44.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:44.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:44.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:44.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:44.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:44.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:44.337+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:44.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:44.669+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:44.669+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:44.669+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:44.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:44.670+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:45.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:45.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:45.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:45.385+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:45.169+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:45.169+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:45.169+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:45.169+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:45.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:45.689+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:45.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:45.689+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:45.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:45.689+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:45.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:45.690+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:46.416 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:46.161+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:46.416 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:46.161+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:46.416 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:46.161+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:46.416 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:46.161+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:46.414+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:46.414+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:46.414+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:46.414+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:46.976 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:46.718+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:46.976 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:46.718+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:46.976 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:46.718+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:46.976 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:46.718+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:47.370 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:26:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:26:46.973+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 40824 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:26:47.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:47.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:47.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:47.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:47.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:47.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:47.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:47.194+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:47.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:47.397+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:47.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:47.397+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:47.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:47.397+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:47.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:47.398+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:47.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:47.689+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:47.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:47.689+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:47.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:47.689+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:47.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:47.690+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:48.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:48.368+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:48.368+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:48.368+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:48.369+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:48.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:48.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:48.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:48.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:48.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:48.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:48.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:48.184+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:48.694+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:48.694+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:48.694+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:48.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:48.694+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:49.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:49.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:49.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:49.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:49.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:49.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:49.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:49.189+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:49.328+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:49.328+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:49.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:49.328+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:49.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:49.328+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:49.664+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:49.664+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:49.664+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:49.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:49.664+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:50.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:50.223+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:50.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:50.223+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:50.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:50.223+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:50.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:50.224+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:50.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:50.307+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:50.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:50.307+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:50.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:50.307+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:50.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:50.307+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:50.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:50.665+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:50.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:50.665+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:50.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:50.665+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:50.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:50.666+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:51.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:51.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:51.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:51.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:51.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:51.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:51.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:51.178+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:51.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:51.343+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:51.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:51.343+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:51.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:51.343+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:51.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:51.344+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:51.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:51.650+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:51.650+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:51.650+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:51.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:51.651+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:52.263 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:26:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:26:51.978+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 40912 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:26:52.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:52.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:52.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:52.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:52.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:52.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:52.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:52.175+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:52.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:52.329+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:52.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:52.329+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:52.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:52.329+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:52.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:52.330+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:52.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:52.635+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:52.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:52.635+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:52.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:52.635+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:52.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:52.636+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:53.168+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:53.168+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:53.168+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:53.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:53.168+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:53.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:53.372+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:53.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:53.372+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:53.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:53.372+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:53.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:53.372+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:53.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:53.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:53.668+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:53.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:53.668+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:54.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:54.215+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:54.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:54.215+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:54.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:54.215+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:54.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:54.216+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:54.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:54.392+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:54.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:54.392+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:54.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:54.392+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:54.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:54.393+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:54.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:54.671+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:54.671+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:54.671+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:54.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:54.672+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:55.667 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:55.248+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:55.667 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:55.248+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:55.667 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:55.248+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:55.667 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:55.248+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:55.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:55.417+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:55.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:55.417+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:55.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:55.417+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:55.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:55.418+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:55.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:55.664+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:55.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:55.664+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:55.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:55.664+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:23.026600+0000 (oldest deadline 2024-09-20T20:16:43.026600+0000) 2024-09-20T20:26:55.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:55.665+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:56.664 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:56.278+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:56.665 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:56.278+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:56.665 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:56.278+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:56.665 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:56.279+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:56.665 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:56.416+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:56.665 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:56.416+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:56.665 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:56.416+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:56.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:56.417+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:56.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:56.662+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:57.308 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:26:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:26:56.984+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 41002 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:26:57.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:57.306+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:57.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:57.306+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:57.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:57.306+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:57.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:57.307+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:57.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:57.435+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:57.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:57.435+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:57.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:57.435+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:57.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:57.435+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:57.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:57.693+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:58.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:58.425+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:58.654 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:58.425+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:58.654 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:58.425+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:22.660969+0000 (oldest deadline 2024-09-20T20:16:42.660969+0000) 2024-09-20T20:26:58.654 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:58.425+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:58.654 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:58.281+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:58.654 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:58.281+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:58.655 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:58.281+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:58.655 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:58.282+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:58.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:58.651+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:26:59.648 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:26:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:26:59.454+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:59.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:59.269+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:59.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:59.269+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:59.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:59.269+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:26:59.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:26:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:26:59.270+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:26:59.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:26:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:26:59.646+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:00.662 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:00.407+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:00.662 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:00.310+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:00.663 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:00.310+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:00.663 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:00.310+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:00.663 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:00.311+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:00.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:00.660+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:01.422+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:01.292+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:01.292+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:01.292+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:01.293+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:01.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:01.696+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:02.255 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:27:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:27:01.989+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 41093 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:27:02.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:02.435+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:02.318+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:02.318+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:02.318+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:02.319+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:02.745+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:03.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:03.352+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:03.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:03.352+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:03.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:03.352+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:03.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:03.353+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:03.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:03.432+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:03.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:03.790+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:04.311+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:04.311+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:04.311+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:04.312+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:04.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:04.416+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:04.747+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:05.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:05.417+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:05.326+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:05.326+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:05.326+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:05.327+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:06.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:05.785+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:06.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:06.389+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:06.362+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:06.362+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:06.362+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:06.363+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:07.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:27:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:27:06.994+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 41185 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:27:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:06.833+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:07.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:07.427+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:07.380+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:07.380+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:07.380+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:07.381+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:07.831+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:08.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:08.446+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:08.382+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:08.382+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:08.382+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:08.383+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:08.849+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:09.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:09.425+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:09.371+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:09.371+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:09.371+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:09.372+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:10.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:09.826+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:10.433+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:10.386+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:10.386+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:10.386+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:10.387+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:11.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:10.876+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:11.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:11.430+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:11.337+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:11.337+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:11.337+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:16:36.842794+0000 (oldest deadline 2024-09-20T20:16:56.842794+0000) 2024-09-20T20:27:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:11.338+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:12.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:27:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:27:11.999+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 41274 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:27:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:11.919+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:12.422+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:12.365+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:13.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:12.884+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:13.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:13.365+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:13.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:13.424+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:13.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:13.917+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:14.320+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:14.408+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:14.930+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:15.333+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:15.458+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:16.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:15.931+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:16.293+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:16.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:16.456+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:17.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:27:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:27:17.005+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 41365 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:27:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:16.955+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:17.480 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:17.336+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:17.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:17.485+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:18.339 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:17.990+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:18.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:18.336+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:18.525+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:19.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:18.957+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:18.957+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:18.957+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:18.957+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:19.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:19.385+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:19.495+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:20.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:20.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:20.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:20.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:20.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:20.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:20.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:20.001+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:20.592 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:20.396+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:20.593 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:20.481+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:21.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:21.044+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:21.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:21.044+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:21.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:21.044+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:21.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:21.045+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:21.719 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:21.446+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:21.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:21.514+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:21.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:21.514+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:21.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:21.514+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:21.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:21.514+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:22.263 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:27:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:27:22.010+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 41453 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:27:22.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:22.088+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:22.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:22.088+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:22.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:22.088+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:22.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:22.088+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:22.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:22.454+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:22.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:22.506+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:22.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:22.506+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:22.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:22.506+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:22.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:22.507+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:23.418 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:23.060+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:23.418 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:23.060+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:23.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:23.060+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:23.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:23.060+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:23.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:23.416+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:23.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:23.539+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:23.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:23.539+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:23.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:23.539+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:23.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:23.539+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:24.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:24.097+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:24.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:24.097+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:24.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:24.097+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:24.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:24.098+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:24.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:24.424+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:24.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:24.493+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:24.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:24.493+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:24.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:24.493+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:24.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:24.494+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:25.437 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:25.105+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:25.437 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:25.105+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:25.437 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:25.105+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:25.437 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:25.105+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:25.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:25.434+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:25.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:25.465+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:25.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:25.465+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:25.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:25.465+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:25.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:25.466+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:26.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:26.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:26.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:26.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:26.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:26.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:26.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:26.062+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:26.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:26.454+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:26.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:26.513+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:26.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:26.513+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:26.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:26.513+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:26.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:26.514+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:27.371 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:27:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:27:27.015+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 41544 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:27:27.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:27.047+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:27.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:27.047+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:27.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:27.047+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:27.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:27.048+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:27.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:27.410+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:27.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:27.509+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:27.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:27.509+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:27.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:27.509+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:27.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:27.510+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:28.407 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:28.059+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:28.407 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:28.059+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:28.407 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:28.059+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:28.407 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:28.060+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:28.405+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:28.554+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:28.554+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:28.554+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:28.555+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:29.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:29.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:29.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:29.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:29.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:29.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:29.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:29.026+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:29.443+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:29.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:29.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:29.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:29.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:29.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:29.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:29.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:29.587+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:30.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:30.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:30.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:30.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:30.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:30.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:30.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:30.030+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:30.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:30.410+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:30.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:30.541+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:30.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:30.541+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:30.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:30.541+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:30.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:30.542+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:31.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:31.011+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:31.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:31.011+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:31.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:31.011+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:31.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:31.012+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:31.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:31.371+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:31.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:31.497+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:31.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:31.497+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:31.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:31.497+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:31.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:31.498+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:32.255 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:27:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:27:32.021+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 41632 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:27:32.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:31.991+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:32.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:31.991+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:32.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:31.991+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:32.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:31.992+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:32.356+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:32.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:32.481+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:32.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:32.481+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:32.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:32.481+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:32.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:32.482+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:32.976+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:32.976+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:32.976+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:32.976+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:33.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:33.394+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:33.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:33.395+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:33.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:33.395+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:33.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:33.395+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:33.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:33.459+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:33.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:33.459+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:33.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:33.459+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:33.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:33.460+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:34.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:33.955+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:34.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:33.955+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:34.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:33.955+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:34.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:33.956+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:34.350+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:34.350+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:34.350+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:34.351+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:34.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:34.417+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:34.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:34.417+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:34.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:34.417+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:34.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:34.418+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:35.361 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:34.994+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:35.362 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:34.994+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:35.362 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:34.994+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:35.362 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:34.995+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:35.359+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:35.359+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:35.359+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:35.360+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:35.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:35.458+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:35.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:35.458+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:35.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:35.458+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:35.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:35.458+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:35.986+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:35.986+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:35.986+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:35.986+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:36.376+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:36.376+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:36.376+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:36.377+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:36.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:36.499+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:36.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:36.499+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:36.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:36.499+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:36.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:36.500+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:37.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:27:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:27:37.026+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 41723 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:27:37.037 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:36.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:37.037 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:36.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:37.037 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:36.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:37.037 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:36.986+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:37.376+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:37.376+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:37.376+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:37.376+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:37.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:37.503+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:37.503+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:37.503+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:37.504+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:38.329 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:37.981+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:38.330 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:37.981+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:38.330 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:37.981+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:38.330 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:37.982+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:38.327+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:38.327+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:38.327+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:38.328+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:38.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:38.487+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:38.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:38.487+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:38.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:38.487+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:38.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:38.488+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:38.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:38.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:38.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:38.934+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:39.506+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:39.506+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:39.506+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:39.506+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:39.315+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:39.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:39.315+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:39.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:39.315+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:39.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:39.316+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:39.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:39.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:39.915+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:39.916+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:40.324+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:40.324+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:40.324+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:40.325+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:40.457+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:40.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:40.457+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:40.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:40.457+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:40.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:40.458+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:40.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:40.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:40.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:41.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:40.959+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:41.290+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:41.290+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:41.290+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:41.290+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:41.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:41.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:41.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:41.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:41.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:41.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:41.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:41.410+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:42.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:27:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:27:42.032+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 41812 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:27:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:41.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:41.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:41.938+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:41.938+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:42.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:42.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:42.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:42.246+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:42.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:42.423+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:42.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:42.423+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:42.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:42.423+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:42.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:42.424+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:42.916+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:42.916+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:42.916+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:42.917+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:43.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:43.273+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:43.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:43.273+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:43.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:43.273+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:43.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:43.274+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:43.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:43.451+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:43.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:43.451+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:43.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:43.451+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:43.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:43.452+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:44.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:43.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:44.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:43.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:44.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:43.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:44.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:43.949+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:44.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:44.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:44.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:44.307+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:44.469+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:44.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:44.469+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:44.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:44.469+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:44.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:44.470+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:45.317 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:44.963+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:45.318 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:44.963+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:45.318 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:44.963+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:45.318 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:44.964+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:45.315+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:45.315+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:45.315+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:45.315+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:45.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:45.430+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:45.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:45.430+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:45.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:45.430+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:45.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:45.430+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:46.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:45.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:45.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:45.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:45.963+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:46.304+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:46.304+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:46.304+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:46.305+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:46.403+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:46.403+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:46.403+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:46.404+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:47.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:27:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:27:47.039+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 41904 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:27:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:46.991+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:46.991+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:46.991+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:46.991+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:47.469 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:47.284+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:47.469 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:47.284+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:47.470 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:47.284+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:47.470 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:47.285+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:47.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:47.437+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:47.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:47.437+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:47.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:47.437+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:47.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:47.438+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:48.310 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:47.991+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:48.310 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:47.991+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:48.311 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:47.991+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:48.311 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:47.992+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:48.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:48.308+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:48.308+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:48.308+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:48.309+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:48.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:48.427+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:48.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:48.427+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:48.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:48.427+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:48.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:48.428+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:49.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:49.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:49.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:49.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:49.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:49.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:49.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:49.010+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:49.339+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:49.339+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:49.339+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:49.340+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:49.402+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:49.402+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:49.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:49.402+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:49.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:49.403+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:50.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:49.991+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:50.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:49.991+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:50.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:49.991+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:50.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:49.992+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:50.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:50.432+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:50.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:50.432+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:50.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:50.432+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:50.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:50.432+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:50.341+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:50.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:50.341+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:50.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:50.341+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:50.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:50.342+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:50.996+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:50.996+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:50.996+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:50.997+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:51.351+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:51.351+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:51.351+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:51.352+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:51.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:51.411+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:51.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:51.411+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:51.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:51.411+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:51.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:51.412+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:52.360 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:27:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:27:52.044+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 41993 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:27:52.361 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:52.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:52.361 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:52.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:52.361 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:52.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:52.361 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:52.017+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:52.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:52.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:52.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:52.365+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:52.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:52.396+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:52.396+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:52.396+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:52.396+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:53.353 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:53.023+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:53.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:53.023+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:53.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:53.023+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:53.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:53.024+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:53.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:53.351+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:53.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:53.351+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:53.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:53.351+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:53.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:53.352+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:53.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:53.391+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:53.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:53.391+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:53.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:53.391+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:53.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:53.392+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:54.385 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:53.978+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:54.385 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:53.978+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:54.385 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:53.979+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:54.385 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:53.979+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:54.396+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:54.396+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:54.396+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:54.397+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:54.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:54.425+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:54.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:54.425+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:54.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:54.425+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:54.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:54.426+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:55.357 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:55.027+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:55.357 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:55.027+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:55.357 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:55.027+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:55.357 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:55.028+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:55.354+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:55.354+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:55.354+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:55.355+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:55.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:55.463+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:55.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:55.463+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:55.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:55.463+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:55.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:55.464+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:56.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:56.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:56.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:56.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:56.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:56.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:56.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:56.026+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:56.374+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:56.374+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:56.374+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:56.375+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:56.434+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:56.434+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:56.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:56.434+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:56.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:56.435+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:57.373 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:27:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:27:57.049+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 42084 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:27:57.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:57.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:57.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:57.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:57.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:57.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:57.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:57.023+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:57.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:57.410+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:57.410+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:57.410+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:57.410+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:57.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:57.438+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:57.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:57.438+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:57.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:57.438+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:57.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:57.439+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:58.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:58.010+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:58.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:58.010+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:58.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:58.010+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:58.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:58.011+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:58.447+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:58.447+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:58.447+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:58.448+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:58.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:58.476+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:58.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:58.476+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:58.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:58.476+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:58.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:58.476+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:59.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:59.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:59.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:59.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:59.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:59.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:27:59.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:59.010+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:27:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:59.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:59.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:59.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:27:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:27:59.461+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:27:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:59.446+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:59.446+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:59.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:59.446+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:27:59.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:27:59.447+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:00.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:59.984+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:00.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:59.984+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:00.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:59.984+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:00.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:27:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:27:59.985+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:00.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:00.474+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:00.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:00.474+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:00.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:00.474+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:00.529 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:00.475+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:00.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:00.451+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:00.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:00.451+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:00.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:00.451+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:00.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:00.451+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:00.957+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:00.957+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:00.957+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:01.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:00.957+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:01.433+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:01.433+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:01.433+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:01.434+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:01.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:01.501+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:01.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:01.501+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:01.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:01.501+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:01.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:01.501+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:02.234 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:28:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:28:02.055+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 42173 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:28:02.234 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:01.965+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:02.234 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:01.965+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:02.234 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:01.965+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:02.234 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:01.965+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:02.457+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:02.457+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:02.457+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:02.458+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:02.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:02.462+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:02.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:02.462+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:02.462+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:02.463+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:02.919+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:02.919+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:02.919+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:02.920+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:03.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:03.480+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:03.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:03.480+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:03.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:03.480+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:03.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:03.481+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:03.906 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:03.490+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:03.906 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:03.490+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:03.906 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:03.490+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:03.906 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:03.491+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:03.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:03.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:03.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:04.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:03.903+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:04.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:04.465+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:04.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:04.465+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:04.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:04.465+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:04.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:04.466+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:04.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:04.465+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:04.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:04.465+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:04.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:04.465+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:04.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:04.466+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:04.868+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:04.868+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:04.868+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:04.869+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:05.467+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:05.467+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:05.467+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:05.467+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:05.433+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:05.433+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:05.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:05.433+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:05.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:05.434+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:05.859+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:05.859+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:05.859+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:05.859+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:06.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:06.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:06.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:06.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:06.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:06.424+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:06.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:06.424+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:06.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:06.391+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:06.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:06.391+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:06.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:06.391+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:06.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:06.392+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:06.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:06.861+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:06.861+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:06.861+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:06.862+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:07.317 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:28:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:28:07.061+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 42265 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:28:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:07.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:07.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:07.397+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:07.397+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:07.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:07.425+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:07.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:07.425+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:07.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:07.425+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:07.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:07.426+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:07.886+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:07.886+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:07.886+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:07.887+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:08.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:08.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:08.441+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:08.442+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:08.404+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:08.404+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:08.404+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:08.405+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:08.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:08.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:08.862+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:08.862+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:09.452+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:09.452+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:09.452+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:09.453+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:09.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:09.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:09.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:09.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:09.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:09.360+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:10.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:09.869+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:10.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:09.869+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:10.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:09.869+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:10.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:09.870+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:10.446+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:10.446+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:10.446+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:10.447+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:10.394+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:10.394+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:10.394+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:10.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:10.394+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:10.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:10.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:10.834+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:10.835+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:11.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:11.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:11.489+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:11.490+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:11.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:11.372+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:11.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:11.372+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:11.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:11.372+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:11.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:11.373+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:12.069 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:11.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:12.070 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:11.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:12.070 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:11.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:12.070 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:11.812+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:12.362 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:28:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:28:12.067+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 42354 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:28:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:12.530+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:12.530+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:12.530+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:12.531+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:12.385+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:12.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:12.385+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:12.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:12.385+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:12.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:12.385+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:12.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:13.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:12.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:13.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:12.840+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:13.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:12.841+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:13.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:13.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:13.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:13.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:13.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:13.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:13.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:13.524+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:13.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:13.419+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:13.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:13.419+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:13.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:13.419+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:13.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:13.420+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:13.850+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:13.850+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:13.850+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:13.851+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:14.542+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:14.542+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:14.542+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:14.543+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:14.390+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:14.390+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:14.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:14.390+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:14.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:14.391+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:14.837+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:14.837+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:14.837+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:14.837+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:15.557+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:15.557+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:15.557+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:15.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:15.557+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:15.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:15.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:15.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:15.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:15.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:15.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:15.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:15.353+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:15.874+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:15.874+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:15.874+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:15.875+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:16.560+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:16.560+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:16.560+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:16.561+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:16.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:16.370+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:16.370+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:16.370+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:16.371+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:17.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:28:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:28:17.073+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 42445 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:28:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:16.870+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:16.870+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:16.870+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:16.870+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:17.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:17.358+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:17.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:17.358+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:17.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:17.358+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:17.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:17.358+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:17.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:17.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:17.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:17.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:17.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:17.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:17.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:17.543+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:17.863+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:17.863+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:17.863+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:17.864+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:18.556+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:18.556+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:18.556+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:18.557+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:18.363+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:18.363+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:18.363+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:18.364+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:18.886+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:18.886+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:18.886+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:18.887+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:19.399+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:19.399+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:19.399+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:19.399+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:19.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:19.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:19.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:19.526+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:19.906+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:19.906+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:19.906+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:19.907+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:20.549 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:20.380+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:20.549 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:20.380+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:20.549 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:20.380+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:20.550 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:20.381+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:20.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:20.547+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:20.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:20.547+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:20.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:20.547+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:20.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:20.547+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:21.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:20.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:20.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:20.930+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:20.931+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:21.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:21.342+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:21.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:21.342+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:21.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:21.342+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:21.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:21.342+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:21.544+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:21.544+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:21.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:21.544+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:21.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:21.545+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:22.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:28:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:28:22.078+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 42534 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:28:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:21.937+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:21.937+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:21.937+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:21.937+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:22.558 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:22.344+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:22.558 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:22.344+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:22.558 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:22.344+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:22.559 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:22.345+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:22.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:22.556+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:22.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:22.556+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:22.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:22.556+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:22.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:22.556+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:23.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:22.941+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:23.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:22.941+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:23.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:22.941+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:23.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:22.942+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:23.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:23.538+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:23.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:23.538+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:23.678 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:23.538+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:23.678 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:23.539+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:23.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:23.321+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:23.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:23.321+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:23.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:23.321+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:23.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:23.322+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:23.893+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:23.893+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:23.893+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:23.893+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:24.517+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:24.517+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:24.517+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:24.517+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:24.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:24.281+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:24.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:24.281+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:24.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:24.281+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:24.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:24.282+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:25.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:24.917+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:25.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:24.917+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:25.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:24.917+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:25.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:24.918+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:25.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:25.480+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:25.480+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:25.480+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:25.480+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:25.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:25.310+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:25.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:25.310+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:25.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:25.310+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:25.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:25.311+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:25.960+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:25.960+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:25.960+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:26.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:25.960+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:26.524+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:26.524+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:26.524+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:26.525+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:26.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:26.318+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:26.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:26.318+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:26.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:26.318+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:26.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:26.318+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:27.339 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:28:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:28:27.084+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 42625 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:28:27.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:27.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:27.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:27.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:27.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:27.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:27.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:27.001+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:27.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:27.547+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:27.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:27.547+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:27.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:27.547+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:27.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:27.547+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:27.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:27.337+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:27.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:27.337+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:27.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:27.337+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:27.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:27.338+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:28.351 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:28.020+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:28.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:28.020+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:28.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:28.020+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:28.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:28.020+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:28.499+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:28.499+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:28.499+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:28.500+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:28.349+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:28.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:28.349+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:28.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:28.349+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:28.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:28.350+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:29.337 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:29.045+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:29.337 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:29.045+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:29.338 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:29.045+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:29.338 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:29.046+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:29.497+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:29.497+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:29.497+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:29.498+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:29.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:29.335+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:29.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:29.335+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:29.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:29.335+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:29.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:29.336+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:30.292+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:30.292+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:30.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:30.292+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:30.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:30.293+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:30.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:30.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:30.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:30.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:30.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:30.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:30.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:30.068+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:30.536+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:30.536+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:30.536+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:30.537+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:31.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:31.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:31.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:31.031+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:31.260+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:31.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:31.260+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:31.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:31.260+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:31.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:31.261+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:31.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:31.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:31.543+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:31.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:31.544+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:32.364 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:28:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:28:32.089+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 42714 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:28:32.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:32.057+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:32.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:32.057+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:32.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:32.057+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:32.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:32.058+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:32.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:32.302+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:32.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:32.302+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:32.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:32.302+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:32.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:32.303+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:32.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:32.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:32.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:32.516+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:33.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:33.056+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:33.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:33.056+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:33.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:33.056+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:33.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:33.057+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:33.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:33.291+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:33.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:33.291+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:33.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:33.291+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:33.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:33.292+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:33.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:33.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:33.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:33.541+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:33.542+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:34.103+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:34.103+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:34.103+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:34.103+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:34.315+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:34.315+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:34.315+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:34.315+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:34.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:34.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:34.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:34.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:34.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:34.563+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:34.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:34.564+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:35.339 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:35.068+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:35.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:35.068+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:35.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:35.068+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:35.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:35.069+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:35.562+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:35.562+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:35.562+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:35.562+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:35.337+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:35.337+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:35.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:35.337+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:35.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:35.338+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:36.322 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:36.024+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:36.322 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:36.024+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:36.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:36.024+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:36.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:36.025+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:36.536+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:36.536+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:36.536+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:36.537+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:36.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:36.320+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:36.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:36.320+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:36.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:36.320+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:36.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:36.320+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:37.333 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:28:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:28:37.095+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 42805 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:28:37.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:37.037+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:37.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:37.037+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:37.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:37.037+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:37.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:37.038+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:37.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:37.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:37.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:37.564+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:37.331+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:37.331+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:37.331+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:37.332+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:38.047+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:38.047+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:38.047+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:38.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:38.048+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:38.599 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:38.288+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:38.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:38.288+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:38.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:38.288+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:38.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:38.289+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:38.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:38.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:38.597+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:38.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:38.597+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:39.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:39.056+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:39.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:39.056+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:39.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:39.056+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:39.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:39.056+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:39.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:39.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:39.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:39.551+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:39.331+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:39.331+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:39.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:39.331+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:39.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:39.332+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:40.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:40.042+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:40.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:40.042+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:40.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:40.042+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:40.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:40.042+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:40.379+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:40.379+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:40.379+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:40.380+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:40.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:40.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:40.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:40.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:40.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:40.582+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:41.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:41.041+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:41.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:41.041+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:41.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:41.041+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:41.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:41.042+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:41.590+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:41.590+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:41.590+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:41.590+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:41.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:41.362+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:41.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:41.362+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:41.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:41.362+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:41.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:41.363+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:42.364 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:28:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:28:42.101+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 42894 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:28:42.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:42.080+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:42.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:42.080+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:42.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:42.080+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:42.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:42.081+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:42.582+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:42.582+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:42.582+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:42.582+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:42.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:42.392+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:42.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:42.392+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:42.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:42.392+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:42.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:42.393+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:43.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:43.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:43.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:43.039+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:43.678 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:43.570+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:43.678 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:43.570+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:43.678 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:43.570+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:43.679 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:43.571+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:43.679 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:43.394+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:43.679 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:43.394+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:43.679 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:43.394+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:43.679 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:43.395+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:44.356 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:44.047+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:44.356 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:44.047+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:44.356 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:44.047+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:44.356 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:44.048+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:44.557+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:44.557+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:44.557+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:44.558+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:44.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:44.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:44.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:44.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:44.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:44.353+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:45.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:45.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:45.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:45.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:45.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:45.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:45.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:45.064+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:45.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:45.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:45.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:45.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:45.385+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:45.596+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:45.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:45.596+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:45.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:45.596+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:45.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:45.596+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:46.407 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:46.054+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:46.407 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:46.054+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:46.408 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:46.054+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:46.408 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:46.054+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:46.404+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:46.404+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:46.404+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:46.404+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:46.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:46.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:46.564+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:46.564+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:47.370 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:28:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:28:47.105+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 42986 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:28:47.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:47.046+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:47.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:47.046+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:47.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:47.046+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:47.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:47.047+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:47.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:47.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:47.554+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:47.555+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:47.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:47.390+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:47.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:47.390+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:47.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:47.390+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:47.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:47.390+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:48.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:48.010+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:48.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:48.010+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:48.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:48.010+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:48.355 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:48.011+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:48.559+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:48.559+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:48.559+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:48.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:48.560+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:48.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:48.351+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:48.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:48.351+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:48.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:48.351+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:48.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:48.352+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:49.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:48.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:49.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:48.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:49.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:48.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:49.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:48.994+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:49.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:49.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:49.532+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:49.532+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:49.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:49.385+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:49.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:49.385+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:49.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:49.385+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:49.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:49.386+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:49.950+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:49.950+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:49.950+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:49.950+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:50.509+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:50.509+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:50.509+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:50.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:50.510+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:50.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:50.378+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:50.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:50.378+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:50.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:50.378+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:50.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:50.378+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:50.972+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:50.972+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:50.972+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:51.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:50.972+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:51.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:51.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:51.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:51.467+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:51.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:51.368+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:51.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:51.368+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:51.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:51.368+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:51.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:51.369+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:52.256 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:28:52.112+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43075 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:28:52.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:52.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:52.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:52.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:52.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:52.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:52.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:52.003+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:52.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:52.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:52.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:52.487+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:52.372+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:52.372+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:52.372+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:52.373+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:53.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:52.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:53.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:52.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:53.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:52.958+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:53.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:52.958+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:53.679 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:53.488+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:53.679 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:53.488+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:53.679 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:53.488+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:53.679 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:53.489+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:53.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:53.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:53.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:53.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:53.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:53.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:53.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:53.372+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:54.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:53.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:54.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:53.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:54.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:53.973+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:54.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:53.973+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:54.535+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:54.535+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:54.535+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:54.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:54.535+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:54.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:54.332+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:54.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:54.332+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:54.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:54.332+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:54.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:54.333+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:55.353 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:54.986+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:55.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:54.986+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:55.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:54.986+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:55.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:54.987+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:55.351+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:55.351+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:55.351+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:55.351+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:55.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:55.539+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:55.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:55.539+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:55.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:55.539+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:55.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:55.539+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:56.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:56.004+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:56.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:56.004+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:56.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:56.004+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:56.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:56.005+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:56.559+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:56.559+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:56.559+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:56.560+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:56.360+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:56.360+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:56.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:56.360+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:56.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:56.360+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:57.354 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:28:57.117+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43163 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:28:57.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:57.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:57.355 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:57.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:57.355 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:57.030+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:57.355 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:57.031+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:57.595+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:57.595+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:57.595+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:57.596+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:57.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:57.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:57.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:57.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:57.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:57.352+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:57.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:57.352+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:58.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:57.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:58.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:57.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:58.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:57.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:58.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:57.994+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:58.631 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:58.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:58.631 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:58.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:58.631 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:58.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:58.631 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:58.372+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:58.628+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:58.628+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:58.628+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:58.629+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:59.386 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:59.018+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:59.387 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:59.018+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:59.387 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:59.018+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:28:59.387 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:28:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:28:59.018+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:28:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:59.584+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:59.584+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:59.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:59.584+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:28:59.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:28:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:28:59.585+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:28:59.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:59.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:59.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:59.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:59.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:59.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:28:59.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:28:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:28:59.385+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:00.385 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:00.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:00.385 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:00.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:00.386 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:00.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:00.386 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:00.052+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:00.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:00.547+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:00.547+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:00.547+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:00.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:00.547+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:00.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:00.383+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:00.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:00.383+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:00.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:00.383+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:00.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:00.384+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:01.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:01.053+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:01.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:01.053+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:01.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:01.053+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:01.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:01.053+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:01.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:01.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:01.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:01.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:01.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:01.410+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:01.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:01.411+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:01.592+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:01.592+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:01.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:01.592+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:01.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:01.592+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:02.257 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:29:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:29:02.122+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43246 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:29:02.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:02.098+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:02.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:02.098+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:02.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:02.098+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:02.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:02.099+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:02.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:02.399+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:02.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:02.399+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:02.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:02.399+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:02.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:02.400+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:02.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:02.582+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:02.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:02.582+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:02.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:02.582+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:02.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:02.583+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:03.384 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:03.091+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:03.384 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:03.091+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:03.384 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:03.091+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:03.384 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:03.092+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:03.680 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:03.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:03.680 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:03.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:03.680 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:03.581+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:03.680 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:03.582+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:03.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:03.381+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:03.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:03.381+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:03.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:03.381+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:03.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:03.382+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:04.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:04.100+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:04.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:04.100+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:04.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:04.100+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:04.368 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:04.101+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:04.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:04.365+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:04.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:04.365+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:04.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:04.365+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:04.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:04.365+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:04.626+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:04.626+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:04.626+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:04.627+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:05.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:05.059+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:05.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:05.059+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:05.337 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:05.059+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:05.337 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:05.059+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:05.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:05.334+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:05.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:05.334+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:05.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:05.334+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:05.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:05.334+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:05.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:05.641+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:05.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:05.641+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:05.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:05.641+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:05.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:05.642+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:06.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:06.014+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:06.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:06.014+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:06.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:06.014+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:06.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:06.015+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:06.667 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:06.287+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:06.667 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:06.287+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:06.667 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:06.287+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:06.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:06.288+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:06.665+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:06.665+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:06.665+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:06.665+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:07.314 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:29:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:29:07.129+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43302 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:29:07.315 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:07.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:07.315 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:07.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:07.315 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:07.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:07.315 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:07.062+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:07.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:07.312+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:07.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:07.312+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:07.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:07.312+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:07.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:07.313+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:07.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:07.675+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:07.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:07.675+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:07.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:07.675+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:07.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:07.676+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:08.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:08.029+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:08.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:08.029+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:08.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:08.029+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:08.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:08.030+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:08.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:08.302+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:08.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:08.302+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:08.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:08.302+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:08.712 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:08.303+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:08.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:08.709+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:08.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:08.709+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:08.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:08.709+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:08.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:08.709+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:09.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:09.054+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:09.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:09.054+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:09.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:09.054+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:09.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:09.055+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:09.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:09.288+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:09.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:09.288+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:09.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:09.288+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:09.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:09.289+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:10.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:09.747+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:10.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:09.747+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:10.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:09.747+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:10.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:09.748+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:10.338 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:10.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:10.338 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:10.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:10.338 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:10.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:10.339 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:10.016+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:10.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:10.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:10.336+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:10.337+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:11.054 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:10.763+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:11.055 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:10.763+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:11.055 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:10.763+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:11.055 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:10.764+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:11.320 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:11.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:11.320 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:11.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:11.320 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:11.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:11.320 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:11.053+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:11.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:11.317+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:11.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:11.317+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:11.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:11.317+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:11.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:11.318+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:12.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:11.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:12.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:11.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:12.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:11.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:12.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:11.733+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:12.297 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:29:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:29:12.134+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43346 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:29:12.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:12.020+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:12.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:12.020+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:12.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:12.020+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:12.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:12.021+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:12.295+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:12.295+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:12.295+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:12.296+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:13.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:12.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:13.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:12.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:13.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:12.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:13.052 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:12.732+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:13.412 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:13.049+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:13.412 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:13.049+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:13.412 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:13.049+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:13.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:13.049+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:13.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:13.247+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:13.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:13.247+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:13.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:13.247+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:13.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:13.248+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:13.694+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:13.694+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:13.694+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:13.695+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:14.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:14.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:14.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:14.026+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:14.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:14.239+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:14.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:14.239+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:14.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:14.239+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:14.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:14.240+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:15.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:14.735+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:15.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:14.735+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:15.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:14.735+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:15.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:14.736+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:15.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:15.018+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:15.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:15.018+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:15.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:15.018+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:15.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:15.018+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:15.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:15.227+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:15.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:15.227+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:15.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:15.227+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:15.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:15.228+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:16.200 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:15.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:16.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:15.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:16.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:15.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:16.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:15.986+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:16.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:15.746+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:16.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:15.746+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:16.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:15.746+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:16.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:15.747+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:16.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:16.198+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:16.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:16.198+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:16.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:16.198+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:16.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:16.199+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:16.713+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:16.713+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:16.713+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:16.714+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:17.372 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:29:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:29:17.140+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43392 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:29:17.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:16.991+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:17.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:16.991+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:17.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:16.991+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:17.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:16.992+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:17.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:17.161+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:17.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:17.161+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:17.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:17.161+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:17.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:17.162+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:18.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:17.745+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:18.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:17.745+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:18.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:17.745+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:18.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:17.745+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:18.003+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:18.003+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:18.003+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:18.004+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:18.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:18.188+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:18.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:18.188+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:18.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:18.188+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:18.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:18.188+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:19.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:18.750+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:19.031 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:18.750+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:19.031 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:18.750+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:19.031 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:18.751+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:19.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:19.028+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:19.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:19.028+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:19.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:19.028+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:19.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:19.029+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:19.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:19.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:19.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:19.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:19.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:19.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:19.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:19.142+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:20.039 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:19.753+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:20.039 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:19.753+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:20.039 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:19.753+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:20.039 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:19.754+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:20.400 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:20.036+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:20.400 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:20.036+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:20.401 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:20.036+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:20.401 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:20.037+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:20.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:20.146+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:20.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:20.146+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:20.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:20.146+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:20.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:20.147+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:21.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:20.719+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:21.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:20.719+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:21.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:20.719+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:21.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:20.720+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:21.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:21.026+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:21.026+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:21.026+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:21.027+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:21.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:21.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:21.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:21.170+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:21.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:21.171+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:22.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:21.763+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:22.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:21.763+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:22.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:21.763+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:22.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:21.764+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:22.358 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:29:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:29:22.145+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43436 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:29:22.358 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:22.029+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:22.358 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:22.029+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:22.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:22.029+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:22.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:22.030+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:22.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:22.175+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:22.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:22.175+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:22.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:22.175+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:22.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:22.176+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:23.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:22.720+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:23.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:22.720+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:23.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:22.720+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:23.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:22.721+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:23.185+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:23.185+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:23.185+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:23.186+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:23.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:23.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:23.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:23.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:23.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:23.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:23.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:23.017+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:23.750+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:23.750+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:23.750+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:23.751+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:24.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:24.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:24.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:24.062+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:24.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:24.185+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:24.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:24.185+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:24.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:24.185+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:24.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:24.186+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:25.069 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:24.765+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:25.069 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:24.765+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:25.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:24.765+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:25.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:24.766+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:25.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:25.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:25.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:25.068+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:25.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:25.186+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:25.186+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:25.186+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:25.187+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:26.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:26.029+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:26.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:26.029+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:26.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:26.029+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:26.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:26.030+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:26.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:25.788+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:26.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:25.788+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:26.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:25.788+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:26.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:25.789+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:26.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:26.150+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:26.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:26.150+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:26.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:26.150+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:26.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:26.151+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:27.118 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:27.023+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:27.118 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:27.023+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:27.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:27.023+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:27.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:27.024+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:27.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:26.835+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:27.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:26.835+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:27.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:26.835+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:27.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:26.836+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:27.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:29:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:29:27.151+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43483 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:29:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:27.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:27.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:27.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:27.116+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:28.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:28.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:28.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:28.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:28.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:28.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:28.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:28.022+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:28.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:27.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:28.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:27.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:28.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:27.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:28.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:27.885+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:28.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:28.134+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:28.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:28.134+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:28.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:28.134+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:28.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:28.135+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:28.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:28.839+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:28.839+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:28.839+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:28.840+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:29.085+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:29.085+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:29.085+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:29.086+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:29.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:29.004+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:29.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:29.004+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:29.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:29.004+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:29.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:29.005+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:30.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:30.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:30.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:30.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:30.071+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:30.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:30.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:30.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:30.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:30.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:30.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:30.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:30.003+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:30.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:29.834+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:30.219 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:29.834+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:30.219 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:29.834+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:30.219 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:29.835+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:31.110 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:30.953+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:31.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:30.953+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:31.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:30.953+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:31.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:30.954+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:31.111 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:30.797+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:31.111 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:30.797+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:31.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:30.797+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:31.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:30.798+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:31.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:31.107+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:31.107+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:31.107+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:31.108+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:32.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:31.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:32.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:31.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:32.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:31.962+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:32.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:31.963+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:32.088 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:31.767+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:32.088 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:31.767+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:32.088 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:31.767+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:32.088 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:31.767+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:32.361 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:29:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:29:32.155+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43527 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:29:32.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:32.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:32.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:32.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:32.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:32.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:32.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:32.084+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:32.790+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:32.790+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:32.790+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:32.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:32.791+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:33.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:32.989+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:33.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:32.989+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:33.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:32.989+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:33.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:32.989+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:33.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:33.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:33.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:33.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:33.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:33.084+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:33.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:33.084+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:33.837+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:33.837+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:33.837+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:33.838+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:33.989+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:33.989+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:33.989+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:33.989+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:34.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:34.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:34.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:34.077+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:35.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:35.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:35.076+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:35.077+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:35.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:34.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:35.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:34.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:35.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:34.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:35.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:34.877+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:35.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:34.972+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:35.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:34.972+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:35.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:34.972+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:35.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:34.973+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:36.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:35.943+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:36.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:35.943+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:36.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:35.943+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:36.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:35.943+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:36.090 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:35.924+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:36.090 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:35.924+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:36.090 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:35.924+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:36.090 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:35.925+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:36.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:36.086+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:36.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:36.086+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:36.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:36.086+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:36.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:36.087+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:37.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:36.979+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:37.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:36.979+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:37.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:36.979+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:37.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:36.979+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:37.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:36.886+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:37.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:36.886+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:37.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:36.886+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:37.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:36.887+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:37.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:37.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:37.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:37.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:37.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:37.120+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:37.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:37.121+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:37.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:29:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:29:37.162+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43573 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:29:38.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:37.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:38.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:37.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:38.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:37.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:38.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:37.967+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:38.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:37.843+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:38.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:37.843+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:38.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:37.843+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:38.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:37.844+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:38.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:38.161+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:38.161+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:38.161+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:38.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:38.161+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:39.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:38.925+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:39.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:38.925+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:39.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:38.925+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:39.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:38.926+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:39.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:38.849+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:39.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:38.849+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:39.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:38.849+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:39.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:38.849+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:39.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:39.123+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:39.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:39.123+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:39.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:39.123+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:39.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:39.123+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:40.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:39.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:40.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:39.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:40.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:39.882+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:40.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:39.883+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:40.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:39.866+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:40.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:39.866+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:40.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:39.866+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:40.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:39.866+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:40.127+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:40.127+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:40.127+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:40.128+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:41.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:40.906+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:41.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:40.906+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:41.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:40.906+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:41.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:40.907+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:41.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:40.875+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:41.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:40.875+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:41.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:40.875+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:41.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:40.875+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:41.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:41.092+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:41.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:41.092+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:41.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:41.092+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:41.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:41.093+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:42.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:41.936+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:42.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:41.936+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:42.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:41.936+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:42.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:41.937+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:42.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:41.901+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:42.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:41.901+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:42.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:41.901+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:42.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:41.901+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:42.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:42.079+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:42.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:42.079+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:42.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:42.079+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:42.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:42.079+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:42.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:29:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:29:42.168+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43617 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:29:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:42.896+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:42.896+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:42.896+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:42.897+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:43.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:42.951+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:43.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:42.951+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:43.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:42.951+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:43.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:42.952+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:43.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:43.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:43.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:43.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:43.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:43.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:43.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:43.051+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:43.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:43.894+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:43.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:43.894+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:43.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:43.894+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:43.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:43.895+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:43.936+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:43.936+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:43.936+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:44.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:43.937+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:44.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:44.063+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:44.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:44.063+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:44.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:44.063+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:44.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:44.063+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:45.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:44.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:45.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:44.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:45.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:44.903+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:45.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:44.904+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:44.889+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:44.889+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:45.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:44.889+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:45.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:44.890+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:45.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:45.021+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:45.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:45.021+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:45.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:45.021+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:45.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:45.022+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:46.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:46.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:46.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:46.010+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:46.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:45.900+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:46.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:45.900+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:46.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:45.900+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:46.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:45.901+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:46.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:45.863+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:46.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:45.863+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:46.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:45.863+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:46.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:45.864+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:47.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:47.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:47.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:47.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:47.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:47.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:47.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:47.051+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:47.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:46.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:47.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:46.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:47.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:46.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:47.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:46.876+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:47.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:46.888+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:47.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:46.888+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:47.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:46.888+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:47.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:46.888+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:47.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:29:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:29:47.174+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43664 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:29:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:47.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:47.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:47.841+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:48.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:47.841+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:48.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:47.859+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:48.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:47.859+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:48.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:47.859+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:48.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:47.860+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:48.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:48.051+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:48.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:48.051+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:48.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:48.051+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:48.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:48.052+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:48.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:48.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:48.858+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:49.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:48.859+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:49.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:48.812+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:49.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:48.812+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:49.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:48.812+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:49.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:48.813+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:49.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:49.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:49.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:49.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:49.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:49.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:49.051+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:49.817+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:49.817+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:49.817+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:49.817+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:50.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:49.824+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:50.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:49.824+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:50.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:49.824+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:50.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:49.825+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:50.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:50.020+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:50.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:50.020+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:50.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:50.020+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:50.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:50.021+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:51.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:50.838+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:51.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:50.838+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:51.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:50.838+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:51.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:50.838+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:51.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:50.862+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:51.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:50.862+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:51.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:50.862+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:51.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:50.863+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:51.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:51.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:51.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:51.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:51.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:51.069+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:51.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:51.070+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:52.052 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:51.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:52.053 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:51.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:52.053 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:51.796+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:52.053 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:51.797+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:52.053 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:51.863+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:52.053 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:51.863+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:52.054 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:51.863+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:52.054 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:51.864+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:52.363 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:29:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:29:52.180+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43708 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:29:52.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:52.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:52.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:52.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:52.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:52.050+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:52.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:52.051+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:53.088 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:52.773+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:53.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:52.773+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:53.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:52.773+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:53.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:52.774+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:53.089 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:52.903+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:53.089 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:52.903+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:53.090 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:52.903+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:53.090 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:52.904+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:53.086+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:53.086+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:53.086+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:53.087+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:53.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:53.885+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:53.885+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:53.885+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:53.885+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:53.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:53.738+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:53.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:53.738+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:53.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:53.738+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:53.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:53.739+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:54.088+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:54.088+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:54.088+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:54.089+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:55.099 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:54.777+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:55.099 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:54.777+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:55.099 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:54.777+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:55.099 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:54.778+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:55.100 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:54.862+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:55.100 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:54.862+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:55.100 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:54.862+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:55.100 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:54.863+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:55.096+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:55.096+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:55.096+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:55.097+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:56.058 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:55.761+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:56.058 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:55.761+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:56.058 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:55.761+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:56.058 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:55.762+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:56.058 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:55.864+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:56.059 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:55.864+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:56.059 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:55.864+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:56.059 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:55.865+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:56.055+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:56.055+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:56.055+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:56.056+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:57.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:56.767+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:57.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:56.767+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:57.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:56.767+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:57.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:56.768+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:57.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:56.866+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:57.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:56.866+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:57.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:56.866+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:57.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:56.867+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:57.370 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:29:57.186+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43754 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:29:57.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:57.026+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:57.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:57.026+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:57.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:57.026+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:57.372 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:57.027+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:57.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:57.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:57.811+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:57.811+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:58.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:57.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:58.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:57.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:58.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:57.876+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:58.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:57.877+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:58.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:57.995+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:58.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:57.995+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:58.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:57.995+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:58.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:57.996+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:58.903+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:58.903+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:58.903+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:29:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:58.903+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:29:59.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:58.778+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:59.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:58.778+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:59.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:58.778+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:29:59.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:58.779+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:29:59.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:58.984+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:59.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:58.984+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:59.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:58.984+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:29:59.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:29:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:29:58.985+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:00.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:59.915+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:00.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:59.915+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:00.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:59.915+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:00.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:29:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:29:59.916+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:59.791+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:00.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:59.791+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:00.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:59.791+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:00.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:29:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:29:59.791+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:00.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:00.029+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:00.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:00.029+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:00.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:00.029+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:00.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:00.030+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:01.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:00.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:01.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:00.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:01.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:00.751+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:01.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:00.752+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:01.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:00.868+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:01.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:00.868+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:01.043 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:00.868+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:01.043 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:00.869+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:01.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:01.039+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:01.039+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:01.039+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:01.040+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:02.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:01.734+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:02.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:01.734+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:02.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:01.734+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:02.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:01.735+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:02.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:01.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:02.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:01.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:02.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:01.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:02.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:01.823+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:02.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:02.011+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:02.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:02.011+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:02.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:02.011+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:02.264 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:30:02.191+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43798 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:30:02.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:02.012+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:03.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:02.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:02.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:02.780+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:03.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:02.780+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:02.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:03.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:02.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:03.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:02.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:03.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:02.967+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:03.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:02.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:03.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:02.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:03.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:02.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:03.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:02.833+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:03.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:03.817+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:03.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:03.817+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:03.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:03.817+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:03.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:03.818+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:03.783+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:03.783+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:03.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:03.783+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:03.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:03.784+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:04.299 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:03.969+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:04.299 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:03.969+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:04.299 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:03.969+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:04.299 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:03.970+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:04.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:04.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:04.790+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:04.791+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:04.805+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:05.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:04.805+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:05.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:04.805+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:05.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:04.806+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:04.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:04.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:04.966+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:04.967+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:05.850+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:05.850+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:05.850+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:05.851+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:05.993+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:05.993+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:05.993+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:05.994+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:06.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:05.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:06.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:05.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:06.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:05.822+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:06.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:05.823+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:07.199 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:06.854+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:07.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:06.854+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:07.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:06.854+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:07.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:06.854+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:07.200 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:06.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:07.200 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:06.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:07.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:06.818+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:07.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:06.819+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:07.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:07.004+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:07.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:07.004+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:07.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:07.004+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:07.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:07.005+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:07.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:30:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:30:07.197+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43845 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:30:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:07.865+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:07.865+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:07.865+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:07.866+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:07.827+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:08.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:07.827+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:08.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:07.827+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:08.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:07.828+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:08.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:08.018+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:08.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:08.018+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:08.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:08.018+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:08.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:08.018+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:09.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:08.805+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:09.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:08.805+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:09.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:08.805+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:09.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:08.806+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:08.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:09.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:08.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:09.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:08.887+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:09.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:08.887+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:09.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:08.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:09.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:08.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:09.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:08.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:09.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:08.989+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:09.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:09.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:09.967+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:09.968+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:10.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:09.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:10.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:09.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:10.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:09.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:10.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:09.934+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:10.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:09.799+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:10.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:09.799+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:10.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:09.799+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:10.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:09.800+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:10.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:10.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:10.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:10.966+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:11.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:10.756+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:11.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:10.756+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:11.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:10.756+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:11.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:10.757+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:11.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:10.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:11.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:10.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:11.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:10.939+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:11.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:10.940+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:12.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:11.934+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:12.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:11.934+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:12.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:11.934+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:12.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:11.935+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:12.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:11.948+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:12.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:11.948+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:12.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:11.948+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:12.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:11.948+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:12.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:11.774+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:12.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:11.774+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:12.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:11.774+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:12.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:11.774+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:12.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:30:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:30:12.203+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43889 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:30:12.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:12.801+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:12.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:12.801+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:12.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:12.801+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:12.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:12.802+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:12.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:12.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:12.985+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:13.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:12.986+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:13.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:12.978+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:13.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:12.978+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:13.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:12.978+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:13.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:12.978+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:13.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:13.836+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:13.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:13.836+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:13.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:13.836+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:13.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:13.836+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:13.997+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:13.997+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:13.997+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:14.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:13.997+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:14.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:13.957+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:14.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:13.957+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:14.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:13.957+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:14.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:13.958+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:14.995+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:14.995+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:14.995+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:14.996+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:15.002+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:15.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:15.002+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:15.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:15.002+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:15.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:15.003+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:14.845+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:14.845+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:14.845+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:15.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:14.846+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:15.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:15.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:15.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:15.877+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:15.878+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:16.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:15.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:16.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:15.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:16.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:15.988+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:16.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:15.989+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:16.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:16.030+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:16.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:16.030+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:16.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:16.030+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:16.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:16.031+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:17.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:16.960+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:17.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:16.960+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:17.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:16.960+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:17.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:16.960+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:17.212 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:16.879+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:17.212 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:16.879+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:17.212 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:16.879+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:17.212 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:16.880+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:17.213 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:17.047+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:17.213 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:17.047+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:17.213 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:17.047+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:17.213 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:17.047+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:17.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:30:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:30:17.208+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43935 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:30:18.000 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:17.891+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:18.000 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:17.891+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:18.000 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:17.891+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:18.000 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:17.892+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:17.996+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:17.996+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:17.996+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:18.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:17.997+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:18.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:18.004+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:18.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:18.004+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:18.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:18.004+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:18.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:18.005+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:19.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:19.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:19.000+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:19.001+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:18.909+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:19.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:18.909+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:19.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:18.909+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:19.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:18.910+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:19.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:19.034+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:19.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:19.034+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:19.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:19.034+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:19.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:19.034+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:19.987+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:19.987+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:19.987+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:19.988+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:20.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:19.886+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:20.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:19.886+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:20.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:19.886+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:20.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:19.887+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:20.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:19.999+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:20.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:19.999+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:20.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:19.999+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:20.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:19.999+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:21.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:20.980+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:21.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:20.980+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:21.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:20.980+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:21.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:20.980+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:21.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:20.906+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:21.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:20.906+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:21.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:20.906+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:21.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:20.907+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:21.004+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:21.004+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:21.004+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:21.005+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:21.954+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:21.954+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:21.954+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:21.954+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:22.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:21.861+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:22.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:21.861+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:22.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:21.861+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:22.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:21.862+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:22.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:22.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:22.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:22.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:22.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:22.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:22.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:22.036+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:22.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:30:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:30:22.214+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 43979 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:30:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:23.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:23.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:23.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:23.037+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:23.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:22.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:23.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:22.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:23.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:22.933+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:23.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:22.934+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:23.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:22.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:23.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:22.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:23.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:22.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:23.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:22.848+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:23.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:23.893+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:23.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:23.893+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:23.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:23.893+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:23.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:23.894+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:24.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:23.990+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:23.990+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:23.990+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:23.990+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:24.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:23.944+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:24.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:23.944+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:24.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:23.944+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:24.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:23.945+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:24.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:24.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:24.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:24.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:24.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:24.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:24.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:24.885+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:25.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:24.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:25.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:24.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:25.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:24.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:25.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:24.967+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:25.035+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:25.035+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:25.035+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:25.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:25.036+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:26.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:25.843+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:26.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:25.843+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:26.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:25.843+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:26.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:25.844+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:26.031+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:26.031+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:26.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:26.031+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:26.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:26.032+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:26.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:25.949+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:26.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:25.949+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:26.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:25.949+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:26.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:25.950+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:27.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:27.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:27.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:27.059+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:26.981+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:27.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:26.981+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:27.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:26.981+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:27.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:26.981+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:27.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:26.827+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:27.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:26.827+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:27.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:26.827+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:27.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:26.828+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:27.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:30:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:30:27.220+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44026 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:30:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:27.824+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:27.824+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:27.824+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:27.825+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:28.014+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:28.014+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:28.014+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:28.015+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:28.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:27.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:28.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:27.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:28.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:27.993+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:28.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:27.994+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:28.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:28.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:28.832+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:28.833+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:29.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:28.992+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:29.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:28.992+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:29.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:28.992+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:29.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:28.993+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:29.049+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:29.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:29.049+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:29.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:29.049+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:29.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:29.050+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:30.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:29.802+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:30.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:29.802+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:30.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:29.802+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:30.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:29.803+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:30.041+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:30.041+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:30.041+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:30.042+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:30.038+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:30.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:30.038+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:30.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:30.038+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:30.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:30.039+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:31.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:30.757+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:31.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:30.757+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:31.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:30.757+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:31.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:30.758+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:31.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:31.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:31.025+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:31.026+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:31.028+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:31.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:31.028+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:31.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:31.028+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:31.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:31.029+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:32.062 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:32.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:32.062 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:32.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:32.062 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:32.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:32.062 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:32.018+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:32.062 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:31.782+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:32.062 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:31.782+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:32.063 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:31.782+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:32.063 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:31.782+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:32.357 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:30:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:30:32.225+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44070 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:30:32.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:32.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:32.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:32.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:32.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:32.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:32.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:32.060+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:33.067+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:33.067+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:33.067+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:33.068+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:33.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:33.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:33.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:33.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:33.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:33.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:33.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:33.007+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:33.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:32.827+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:33.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:32.827+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:33.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:32.827+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:33.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:32.827+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:33.835+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:33.835+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:33.835+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:33.835+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:34.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:34.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:34.115+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:34.115+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:33.992+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:34.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:33.992+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:34.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:33.992+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:34.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:33.993+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:35.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:35.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:35.082 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:35.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:35.082 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:35.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:35.082 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:35.007+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:35.082 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:34.846+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:35.082 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:34.846+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:35.082 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:34.846+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:35.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:34.847+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:35.083 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:35.079+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:35.083 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:35.079+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:35.083 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:35.079+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:35.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:35.080+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:36.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:35.893+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:36.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:35.893+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:36.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:35.893+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:36.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:35.893+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:36.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:36.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:36.077+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:36.078+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:36.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:36.057+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:36.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:36.057+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:36.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:36.057+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:36.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:36.057+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:37.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:37.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:37.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:37.068+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:36.850+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:37.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:36.850+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:37.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:36.850+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:37.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:36.851+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:37.040+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:37.040+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:37.040+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:37.041+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:37.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:30:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:30:37.231+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44116 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:30:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:38.107+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:38.107+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:38.107+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:38.108+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:38.019+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:38.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:38.019+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:38.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:38.019+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:38.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:38.020+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:38.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:37.891+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:38.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:37.891+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:38.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:37.891+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:38.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:37.892+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:39.084+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:39.084+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:39.084+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:39.085+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:39.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:39.033+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:39.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:39.033+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:39.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:39.033+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:39.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:39.034+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:39.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:38.856+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:39.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:38.856+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:39.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:38.856+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:39.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:38.857+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:40.073+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:40.073+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:40.073+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:40.074+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:40.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:39.882+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:40.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:39.882+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:40.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:39.882+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:40.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:39.882+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:40.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:40.010+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:40.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:40.010+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:40.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:40.010+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:40.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:40.010+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:41.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:40.852+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:41.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:40.852+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:41.065 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:40.852+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:41.065 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:40.852+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:41.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:40.965+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:41.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:40.965+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:41.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:40.965+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:41.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:40.966+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:41.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:41.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:41.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:41.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:41.062+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:42.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:42.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:42.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:42.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:42.022+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:42.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:41.856+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:42.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:41.856+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:42.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:41.856+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:42.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:41.857+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:42.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:41.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:42.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:41.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:42.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:41.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:42.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:41.927+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:42.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:30:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:30:42.237+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44160 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:30:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:43.041+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:43.041+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:43.041+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:43.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:43.042+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:43.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:42.813+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:43.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:42.813+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:43.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:42.813+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:43.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:42.814+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:43.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:42.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:43.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:42.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:43.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:42.880+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:43.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:42.881+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:43.790+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:43.790+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:43.790+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:43.790+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:43.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:43.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:43.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:43.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:43.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:43.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:43.900+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:44.053+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:44.053+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:44.053+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:44.054+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:45.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:45.020+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:45.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:45.020+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:45.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:45.020+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:45.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:45.021+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:44.931+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:45.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:44.931+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:45.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:44.931+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:45.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:44.932+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:45.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:44.823+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:45.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:44.823+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:45.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:44.823+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:45.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:44.824+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:46.055+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:46.055+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:46.055+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:46.055+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:46.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:45.838+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:46.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:45.838+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:46.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:45.838+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:46.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:45.839+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:45.969+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:45.969+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:45.969+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:46.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:45.970+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:47.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:46.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:47.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:46.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:47.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:46.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:47.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:46.822+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:47.084 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:46.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:47.084 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:46.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:47.084 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:46.959+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:47.084 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:46.960+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:47.374 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:30:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:30:47.242+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44207 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:30:47.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:47.081+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:47.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:47.081+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:47.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:47.081+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:47.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:47.081+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:48.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:47.843+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:48.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:47.843+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:48.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:47.843+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:48.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:47.844+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:48.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:47.977+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:48.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:47.977+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:48.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:47.977+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:48.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:47.978+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:48.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:48.110+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:48.110+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:48.110+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:48.111+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:49.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:48.841+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:49.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:48.841+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:49.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:48.841+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:49.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:48.841+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:49.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:48.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:49.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:48.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:49.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:48.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:49.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:48.981+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:49.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:49.138+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:49.138+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:49.138+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:49.139+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:50.104 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:49.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:50.104 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:49.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:50.104 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:49.847+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:50.105 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:49.848+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:50.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:49.948+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:50.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:49.948+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:50.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:49.948+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:50.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:49.948+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:50.172+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:50.172+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:50.172+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:50.172+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:51.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:50.857+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:51.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:50.857+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:51.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:50.857+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:51.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:50.857+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:51.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:50.945+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:51.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:50.945+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:51.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:50.945+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:51.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:50.946+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:51.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:51.126+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:51.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:51.126+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:51.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:51.126+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:51.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:51.126+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:52.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:51.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:52.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:51.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:52.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:51.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:52.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:51.885+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:52.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:51.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:52.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:51.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:52.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:51.957+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:52.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:51.958+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:52.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:30:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:30:52.248+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44251 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:30:52.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:52.155+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:52.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:52.155+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:52.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:52.155+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:52.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:52.155+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:53.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:52.920+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:53.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:52.920+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:53.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:52.920+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:53.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:52.921+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:53.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:52.929+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:53.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:52.929+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:53.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:52.929+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:53.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:52.930+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:53.130+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:53.130+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:53.130+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:53.131+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:53.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:53.885+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:53.885+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:53.885+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:53.886+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:53.894+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:53.894+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:53.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:53.894+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:53.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:53.895+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:54.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:54.164+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:54.164+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:54.164+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:54.165+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:55.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:54.875+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:55.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:54.875+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:55.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:54.875+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:55.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:54.876+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:55.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:54.919+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:55.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:54.919+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:55.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:54.919+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:55.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:54.920+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:55.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:55.177+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:55.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:55.177+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:55.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:55.177+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:55.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:55.178+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:55.912+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:55.912+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:55.912+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:56.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:55.913+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:55.946+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:55.946+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:55.946+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:55.947+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:56.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:56.135+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:56.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:56.135+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:56.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:56.135+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:56.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:56.136+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:57.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:57.088+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:57.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:57.088+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:57.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:57.088+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:57.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:57.089+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:57.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:56.901+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:57.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:56.901+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:57.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:56.901+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:57.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:56.901+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:57.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:56.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:57.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:56.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:57.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:56.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:57.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:56.981+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:57.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:30:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:30:57.254+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44297 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:30:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:57.937+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:57.937+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:57.937+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:57.938+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:58.042+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:58.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:58.042+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:58.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:58.042+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:58.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:58.043+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:58.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:57.855+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:58.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:57.855+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:58.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:57.855+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:58.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:57.856+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:58.983+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:58.983+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:58.983+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:30:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:30:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:58.984+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:30:59.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:59.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:59.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:59.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:59.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:59.022+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:30:59.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:30:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:30:59.022+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:30:59.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:58.905+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:59.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:58.905+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:59.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:58.905+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:30:59.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:58.905+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:00.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:00.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:00.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:00.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:00.062+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:00.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:59.927+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:00.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:59.927+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:00.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:59.927+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:00.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:30:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:30:59.928+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:00.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:59.999+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:00.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:59.999+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:00.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:30:59.999+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:00.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:00.000+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:01.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:00.957+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:01.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:00.957+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:01.037 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:00.957+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:01.037 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:00.957+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:01.033+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:01.033+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:01.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:01.033+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:01.033+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:01.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:01.097+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:01.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:01.097+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:01.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:01.097+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:01.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:01.098+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:02.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:01.988+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:02.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:01.988+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:02.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:01.988+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:02.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:01.989+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:02.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:31:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:31:02.259+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44341 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:31:02.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:02.075+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:02.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:02.075+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:02.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:02.075+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:02.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:02.076+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:02.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:02.075+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:02.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:02.075+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:02.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:02.075+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:02.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:02.076+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:03.038+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:03.038+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:03.038+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:03.039+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:03.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:03.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:03.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:03.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:03.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:03.062+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:03.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:03.063+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:03.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:03.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:03.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:03.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:03.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:03.058+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:03.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:03.058+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:04.043+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:04.043+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:04.043+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:04.044+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:04.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:04.054+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:04.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:04.054+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:04.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:04.054+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:04.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:04.054+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:04.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:04.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:04.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:04.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:04.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:04.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:04.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:04.010+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:05.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:05.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:05.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:05.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:05.006+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:05.093+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:05.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:05.093+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:05.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:05.093+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:05.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:05.094+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:04.965+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:04.965+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:04.965+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:04.965+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:06.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:05.996+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:06.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:05.996+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:06.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:05.996+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:06.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:05.997+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:06.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:06.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:06.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:06.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:06.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:06.105+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:06.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:06.105+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:06.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:06.005+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:06.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:06.005+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:06.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:06.005+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:06.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:06.006+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:07.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:07.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:07.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:07.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:07.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:07.002+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:07.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:07.002+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:07.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:07.066+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:07.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:07.066+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:07.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:07.066+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:07.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:07.067+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:06.990+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:06.990+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:06.990+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:06.990+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:07.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:31:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:31:07.266+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44388 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:31:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:08.112+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:08.112+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:08.112+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:08.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:08.113+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:08.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:08.043+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:08.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:08.043+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:08.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:08.043+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:08.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:08.043+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:08.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:07.969+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:08.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:07.969+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:08.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:07.969+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:08.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:07.969+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:09.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:09.095+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:09.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:09.095+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:09.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:09.095+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:09.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:09.095+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:09.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:09.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:09.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:09.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:09.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:09.038+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:09.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:09.039+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:09.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:08.943+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:09.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:08.943+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:09.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:08.943+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:09.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:08.944+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:10.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:10.013+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:10.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:10.013+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:10.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:10.013+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:10.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:10.014+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:10.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:10.071+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:10.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:10.071+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:10.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:10.071+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:10.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:10.072+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:10.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:09.968+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:10.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:09.968+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:10.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:09.968+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:10.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:09.969+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:11.037 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:10.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:11.037 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:10.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:11.037 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:10.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:11.037 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:10.935+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:11.034+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:11.034+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:11.034+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:11.034+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:11.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:11.050+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:11.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:11.050+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:11.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:11.050+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:11.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:11.051+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:12.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:12.089+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:12.089+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:12.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:12.089+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:11.995+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:12.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:11.995+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:12.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:11.995+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:12.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:11.996+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:12.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:11.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:12.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:11.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:12.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:11.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:12.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:11.927+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:12.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:31:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:31:12.272+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44432 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:31:13.061 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:12.998+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:13.062 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:12.998+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:13.062 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:12.998+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:13.062 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:12.998+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:13.062 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:12.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:13.062 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:12.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:13.062 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:12.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:13.063 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:12.929+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:13.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:13.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:13.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:13.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:13.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:13.059+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:13.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:13.060+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:13.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:13.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:13.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:13.916+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:14.039+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:14.039+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:14.039+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:14.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:14.040+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:14.038+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:14.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:14.038+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:14.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:14.038+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:14.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:14.039+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:15.008+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:15.008+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:15.008+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:15.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:15.009+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:15.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:15.010+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:15.010+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:15.010+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:15.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:15.010+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:15.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:14.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:15.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:14.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:15.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:14.930+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:15.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:14.931+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:15.980+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:15.980+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:15.980+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:15.981+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:16.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:16.045+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:16.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:16.045+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:16.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:16.045+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:16.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:16.046+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:16.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:15.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:16.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:15.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:16.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:15.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:16.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:15.917+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:17.027+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:17.027+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:17.027+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:17.028+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:17.049+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:17.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:17.049+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:17.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:17.049+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:17.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:17.049+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:17.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:16.947+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:17.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:16.947+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:17.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:16.947+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:17.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:16.948+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:17.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:31:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:31:17.277+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44478 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:31:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:18.035+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:18.035+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:18.035+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:18.036+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:18.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:18.011+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:18.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:18.011+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:18.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:18.011+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:18.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:18.012+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:18.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:17.938+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:18.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:17.938+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:18.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:17.938+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:18.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:17.939+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:19.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:19.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:19.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:19.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:19.018+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:18.935+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:19.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:18.935+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:19.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:18.935+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:19.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:18.936+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:19.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:18.994+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:19.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:18.994+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:19.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:18.994+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:19.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:18.995+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:20.044+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:20.044+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:20.044+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:20.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:20.045+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:20.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:20.025+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:20.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:20.025+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:20.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:20.025+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:20.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:20.026+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:20.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:19.952+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:20.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:19.952+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:20.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:19.952+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:20.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:19.952+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:21.026+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:21.026+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:21.026+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:21.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:21.027+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:20.912+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:21.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:20.912+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:21.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:20.912+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:21.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:20.912+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:21.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:21.060+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:21.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:21.060+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:21.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:21.060+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:21.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:21.061+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:22.073+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:22.073+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:22.073+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:22.074+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:22.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:21.994+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:22.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:21.994+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:22.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:21.994+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:22.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:21.995+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:22.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:21.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:22.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:21.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:22.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:21.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:22.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:21.908+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:22.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:31:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:31:22.284+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44522 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:31:23.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:22.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:23.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:22.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:23.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:22.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:23.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:22.927+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:23.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:23.008+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:23.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:23.008+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:23.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:23.008+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:23.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:23.008+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:23.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:23.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:23.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:23.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:23.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:23.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:23.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:23.089+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:23.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:23.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:23.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:23.924+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:23.995+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:24.561 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:23.995+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:24.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:23.995+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:24.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:23.996+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:24.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:24.076+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:24.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:24.076+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:24.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:24.076+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:24.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:24.077+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:24.908+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:24.908+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:24.908+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:24.908+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:25.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:24.967+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:25.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:24.967+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:25.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:24.967+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:25.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:24.967+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:25.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:25.095+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:25.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:25.095+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:25.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:25.095+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:25.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:25.095+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:26.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:25.934+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:26.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:25.934+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:26.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:25.934+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:26.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:25.934+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:26.084 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:25.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:26.084 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:25.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:26.085 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:25.887+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:26.085 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:25.888+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:26.352 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:26.081+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:26.352 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:26.081+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:26.352 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:26.081+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:26.353 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:26.082+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:27.093+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:27.093+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:27.093+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:27.094+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:27.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:26.943+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:27.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:26.943+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:27.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:26.943+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:27.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:26.944+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:27.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:26.898+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:27.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:26.898+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:27.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:26.898+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:27.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:26.898+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:31:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:31:27.289+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44569 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:31:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:27.986+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:27.986+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:27.986+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:28.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:27.987+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:28.090+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:28.090+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:28.090+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:28.091+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:27.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:27.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:28.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:27.899+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:28.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:27.900+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:29.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:29.032+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:29.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:29.032+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:29.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:29.032+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:29.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:29.032+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:29.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:29.134+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:29.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:29.134+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:29.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:29.134+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:29.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:29.135+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:29.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:28.925+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:29.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:28.925+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:29.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:28.925+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:29.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:28.926+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:30.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:30.029+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:30.029+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:30.029+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:30.030+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:30.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:30.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:30.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:30.088+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:30.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:30.089+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:30.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:29.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:30.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:29.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:30.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:29.941+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:30.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:29.942+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:31.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:30.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:31.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:30.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:31.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:30.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:31.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:30.928+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:31.033+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:31.033+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:31.033+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:31.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:31.034+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:31.119+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:31.119+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:31.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:31.119+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:31.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:31.120+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:32.046+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:32.046+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:32.046+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:32.046+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:32.163+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:32.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:32.163+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:32.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:32.163+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:32.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:32.164+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:32.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:31.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:32.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:31.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:32.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:31.916+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:32.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:31.917+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:32.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:31:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:31:32.295+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44613 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:31:33.071 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:32.949+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:33.072 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:32.949+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:33.072 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:32.949+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:33.072 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:32.949+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:33.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:33.069+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:33.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:33.069+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:33.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:33.069+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:33.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:33.070+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:33.120+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:33.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:33.120+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:33.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:33.120+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:33.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:33.120+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:34.087+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:34.087+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:34.087+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:34.088+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:34.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:34.144+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:34.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:34.144+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:34.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:34.144+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:34.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:34.144+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:33.986+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:33.986+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:34.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:33.986+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:34.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:33.987+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:35.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:35.134+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:35.134+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:35.134+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:35.135+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:35.101+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:35.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:35.101+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:35.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:35.101+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:35.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:35.101+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:34.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:34.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:34.988+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:34.988+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:36.133+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:36.133+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:36.133+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:36.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:36.134+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:36.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:36.135+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:36.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:36.135+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:36.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:36.135+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:36.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:36.136+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:35.946+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:35.946+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:36.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:35.946+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:36.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:35.947+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:37.091+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:37.091+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:37.091+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:37.092+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:37.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:37.109+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:37.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:37.109+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:37.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:37.109+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:37.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:37.110+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:36.950+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:36.950+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:37.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:36.950+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:37.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:36.950+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:37.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:31:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:31:37.301+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44659 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:31:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:38.121+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:38.121+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:38.121+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:38.121+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:38.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:38.099+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:38.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:38.099+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:38.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:38.099+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:38.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:38.100+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:38.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:37.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:38.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:37.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:38.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:37.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:38.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:37.905+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:39.127+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:39.127+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:39.127+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:39.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:39.127+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:39.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:39.093+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:39.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:39.093+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:39.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:39.093+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:39.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:39.094+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:39.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:38.937+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:39.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:38.937+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:39.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:38.937+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:39.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:38.938+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:40.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:40.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:40.109+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:40.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:40.109+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:40.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:40.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:40.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:40.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:40.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:40.106+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:40.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:40.107+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:40.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:39.961+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:40.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:39.961+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:40.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:39.961+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:40.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:39.961+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:41.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:40.973+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:41.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:40.973+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:41.108 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:40.973+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:41.108 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:40.974+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:41.125+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:41.125+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:41.125+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:41.126+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:41.104+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:41.104+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:41.104+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:41.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:41.105+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:41.926+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:41.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:41.927+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:41.928+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:42.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:31:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:31:42.306+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44703 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:31:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:42.149+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:42.149+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:42.149+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:42.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:42.150+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:42.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:42.155+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:42.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:42.155+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:42.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:42.155+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:42.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:42.156+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:43.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:42.882+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:43.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:42.882+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:43.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:42.882+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:43.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:42.883+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:43.155+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:43.155+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:43.155+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:43.155+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:43.108+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:43.108+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:43.108+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:43.109+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:43.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:43.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:43.855+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:43.855+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:44.110+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:44.110+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:44.110+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:44.110+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:44.062+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:44.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:44.062+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:44.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:44.062+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:44.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:44.062+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:45.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:45.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:45.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:45.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:45.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:45.013+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:45.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:45.013+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:45.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:44.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:45.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:44.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:45.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:44.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:45.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:44.887+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:45.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:45.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:45.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:45.061+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:45.062+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:46.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:46.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:46.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:46.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:46.067+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:46.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:46.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:46.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:46.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:46.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:46.002+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:46.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:46.003+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:45.923+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:45.923+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:45.923+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:46.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:45.924+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:47.047+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:47.047+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:47.047+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:47.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:47.047+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:47.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:47.033+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:47.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:47.033+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:47.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:47.033+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:47.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:47.034+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:47.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:46.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:47.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:46.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:47.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:46.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:47.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:46.929+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:47.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:31:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:31:47.312+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44750 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:31:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:48.093+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:48.093+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:48.093+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:48.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:48.094+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:48.073+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:48.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:48.073+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:48.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:48.073+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:48.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:48.074+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:48.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:47.892+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:48.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:47.892+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:48.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:47.892+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:48.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:47.893+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:49.082+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:49.082+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:49.082+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:49.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:49.083+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:49.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:49.026+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:49.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:49.026+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:49.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:49.026+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:49.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:49.026+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:48.929+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:48.929+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:48.929+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:49.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:48.929+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:50.023+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:50.023+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:50.023+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:50.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:50.024+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:50.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:50.048+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:50.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:50.048+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:50.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:50.048+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:50.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:50.049+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:50.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:49.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:50.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:49.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:50.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:49.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:50.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:49.886+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:51.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:50.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:51.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:50.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:51.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:50.886+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:51.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:50.887+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:51.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:51.003+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:51.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:51.003+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:51.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:51.003+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:51.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:51.004+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:51.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:51.094+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:51.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:51.094+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:51.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:51.094+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:51.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:51.094+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:51.992+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:51.992+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:52.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:51.992+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:52.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:51.993+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:52.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:51.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:52.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:51.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:52.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:51.915+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:52.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:51.916+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:52.219 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:52.134+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:52.219 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:52.134+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:52.219 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:52.134+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:52.219 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:52.134+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:52.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:31:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:31:52.318+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44794 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:31:53.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:52.969+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:53.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:52.969+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:53.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:52.969+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:53.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:52.970+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:53.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:52.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:53.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:52.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:53.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:52.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:53.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:52.918+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:53.140+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:53.140+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:53.140+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:53.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:53.140+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:54.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:53.969+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:54.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:53.969+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:54.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:53.969+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:54.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:53.970+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:54.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:54.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:54.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:54.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:54.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:54.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:54.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:54.187+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:54.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:53.950+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:54.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:53.950+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:54.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:53.950+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:54.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:53.951+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:54.974+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:54.974+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:54.974+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:55.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:54.975+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:55.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:54.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:54.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:54.907+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:55.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:54.908+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:55.236+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:55.236+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:55.236+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:55.237+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:56.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:56.024+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:56.024+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:56.024+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:56.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:56.024+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:56.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:55.906+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:55.906+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:55.906+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:56.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:55.907+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:56.213+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:56.213+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:56.213+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:56.214+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:57.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:57.007+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:57.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:57.007+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:57.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:57.007+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:57.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:57.008+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:57.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:56.861+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:57.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:56.861+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:57.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:56.861+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:57.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:56.862+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:57.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:31:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:31:57.325+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44840 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:31:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:57.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:57.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:57.178+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:57.178+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:58.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:58.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:58.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:58.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:58.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:58.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:58.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:58.009+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:58.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:57.843+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:58.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:57.843+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:58.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:57.843+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:58.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:57.844+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:58.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:58.137+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:58.137+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:58.137+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:58.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:58.138+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:59.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:59.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:59.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:59.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:59.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:59.009+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:31:59.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:31:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:31:59.009+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:31:59.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:58.834+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:59.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:58.834+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:59.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:58.834+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:31:59.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:58.835+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:31:59.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:59.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:59.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:59.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:31:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:31:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:31:59.147+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:00.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:00.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:00.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:00.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:00.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:00.001+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:00.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:00.001+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:00.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:59.820+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:00.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:59.820+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:00.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:59.820+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:00.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:31:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:31:59.820+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:00.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:00.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:00.158+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:00.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:00.159+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:01.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:00.969+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:01.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:00.969+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:01.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:00.969+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:01.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:00.970+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:01.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:00.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:01.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:00.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:01.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:00.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:01.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:00.784+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:01.166+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:01.166+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:01.166+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:01.167+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:02.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:01.949+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:02.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:01.949+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:02.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:01.949+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:02.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:01.950+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:02.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:01.746+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:02.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:01.746+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:02.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:01.746+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:02.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:01.747+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:02.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:32:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:32:02.330+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44884 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:32:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:02.181+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:02.181+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:02.181+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:02.182+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:02.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:02.711+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:02.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:02.711+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:02.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:02.711+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:02.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:02.712+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:02.998+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:02.998+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:02.998+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:02.999+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:03.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:03.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:03.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:03.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:03.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:03.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:03.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:03.228+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:03.722+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:03.722+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:03.722+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:03.722+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:04.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:03.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:04.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:03.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:04.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:03.971+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:04.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:03.972+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:04.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:04.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:04.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:04.222+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:04.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:04.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:04.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:05.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:04.948+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:04.758+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:04.758+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:04.758+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:05.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:04.759+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:05.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:05.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:05.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:05.221+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:05.222+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:05.949+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:05.949+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:05.949+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:06.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:05.949+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:05.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:05.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:05.806+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:05.807+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:06.266+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:06.266+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:06.266+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:06.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:06.267+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:06.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:06.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:07.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:06.948+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:07.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:06.949+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:07.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:06.760+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:07.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:06.760+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:07.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:06.760+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:07.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:06.761+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:07.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:32:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:32:07.336+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44931 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:32:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:07.252+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:07.252+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:07.252+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:07.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:07.253+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:07.924+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:07.924+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:07.924+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:07.924+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:07.810+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:08.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:07.810+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:08.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:07.810+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:08.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:07.810+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:08.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:08.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:08.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:08.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:08.284+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:08.883+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:08.883+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:08.883+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:09.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:08.884+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:08.766+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:09.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:08.766+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:09.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:08.766+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:09.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:08.767+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:09.243+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:09.243+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:09.243+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:09.244+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:10.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:09.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:10.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:09.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:10.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:09.855+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:10.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:09.856+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:09.790+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:09.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:10.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:09.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:10.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:09.791+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:10.285+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:10.285+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:10.285+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:10.286+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:10.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:10.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:10.879+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:11.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:10.880+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:10.743+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:10.743+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:11.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:10.743+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:11.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:10.743+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:11.326+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:11.326+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:11.326+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:11.326+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:11.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:11.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:11.875+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:12.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:11.876+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:12.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:11.721+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:12.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:11.721+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:12.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:11.721+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:12.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:11.722+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:12.707 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:32:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:32:12.342+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 44975 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:32:12.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:12.293+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:12.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:12.293+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:12.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:12.293+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:12.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:12.294+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:12.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:12.704+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:12.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:12.704+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:12.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:12.704+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:12.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:12.705+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:12.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:12.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:12.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:12.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:12.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:12.872+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:12.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:12.872+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:13.683 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:13.328+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:13.684 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:13.328+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:13.684 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:13.328+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:13.684 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:13.329+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:13.901+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:13.901+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:13.901+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:13.902+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:13.681+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:13.681+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:13.681+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:13.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:13.682+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:14.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:14.280+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:14.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:14.280+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:14.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:14.280+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:14.669 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:14.281+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:14.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:14.910+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:14.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:14.910+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:14.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:14.910+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:14.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:14.910+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:14.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:14.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:14.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:14.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:14.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:14.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:14.667+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:15.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:15.305+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:15.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:15.305+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:15.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:15.305+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:15.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:15.305+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:15.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:15.639+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:15.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:15.639+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:15.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:15.639+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:15.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:15.640+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:16.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:15.946+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:15.946+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:15.946+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:16.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:15.947+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:16.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:16.348+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:16.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:16.348+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:16.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:16.348+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:16.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:16.349+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:16.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:16.617+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:16.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:16.617+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:16.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:16.617+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:16.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:16.618+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:16.937+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:16.937+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:16.937+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:17.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:16.937+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:17.472 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:32:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:32:17.348+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45021 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:32:17.473 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:17.374+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:17.473 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:17.374+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:17.473 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:17.374+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:17.473 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:17.375+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:17.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:17.595+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:17.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:17.595+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:17.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:17.595+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:17.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:17.596+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:18.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:17.929+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:17.929+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:17.929+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:18.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:17.930+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:18.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:18.621+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:18.621+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:18.621+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:18.622+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:18.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:18.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:18.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:18.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:18.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:18.424+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:19.434 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:18.979+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:19.434 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:18.979+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:19.434 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:18.979+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:19.434 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:18.980+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:19.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:19.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:19.431+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:19.432+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:19.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:19.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:19.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:19.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:19.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:19.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:19.667+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:20.278 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:20.013+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:20.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:20.013+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:20.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:20.013+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:20.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:20.014+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:20.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:20.453+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:20.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:20.453+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:20.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:20.453+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:20.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:20.454+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:20.711+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:20.711+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:20.711+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:20.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:20.712+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:21.008+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:21.008+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:21.008+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:21.009+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:21.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:21.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:21.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:21.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:21.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:21.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:21.741 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:21.468+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:22.034 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:21.737+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:22.034 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:21.737+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:22.034 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:21.737+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:22.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:21.738+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:22.356 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:22.032+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:22.357 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:22.032+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:22.357 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:22.032+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:22.357 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:22.032+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:22.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:32:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:32:22.354+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45065 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:32:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:22.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:22.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:22.515+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:22.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:22.516+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:23.081 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:22.731+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:23.081 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:22.731+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:23.081 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:22.731+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:23.082 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:22.732+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:23.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:23.079+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:23.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:23.079+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:23.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:23.079+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:23.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:23.079+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:23.739 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:23.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:23.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:23.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:23.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:23.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:23.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:23.483+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:24.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:23.737+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:24.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:23.737+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:24.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:23.737+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:24.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:23.738+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:24.055+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:24.055+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:24.055+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:24.056+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:24.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:24.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:24.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:24.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:24.469+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:24.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:24.695+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:24.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:24.695+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:24.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:24.695+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:24.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:24.696+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:25.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:25.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:25.017+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:25.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:25.018+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:25.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:25.470+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:25.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:25.470+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:25.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:25.470+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:25.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:25.471+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:25.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:25.678+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:25.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:25.678+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:25.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:25.678+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:25.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:25.678+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:26.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:26.057+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:26.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:26.057+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:26.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:26.057+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:26.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:26.058+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:26.447+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:26.447+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:26.447+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:26.448+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:26.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:26.629+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:26.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:26.629+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:26.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:26.629+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:26.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:26.630+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:27.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:27.086+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:27.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:27.086+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:27.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:27.086+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:27.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:27.087+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:27.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:32:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:32:27.361+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45112 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:32:27.662 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:27.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:27.663 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:27.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:27.663 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:27.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:27.663 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:27.483+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:27.660+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:27.660+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:27.660+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:27.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:27.660+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:28.448 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:28.042+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:28.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:28.042+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:28.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:28.042+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:28.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:28.043+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:28.446+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:28.446+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:28.446+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:28.447+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:28.645+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:28.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:28.645+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:28.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:28.645+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:28.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:28.646+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:29.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:28.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:29.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:28.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:29.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:28.997+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:29.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:28.997+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:29.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:29.476+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:29.476+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:29.476+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:29.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:29.477+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:29.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:29.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:29.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:29.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:29.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:29.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:29.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:29.597+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:30.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:29.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:30.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:29.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:30.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:29.966+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:30.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:29.967+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:30.564 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:30.494+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:30.564 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:30.494+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:30.564 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:30.494+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:30.564 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:30.495+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:30.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:30.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:30.865 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:30.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:30.865 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:30.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:30.865 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:30.562+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:31.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:30.977+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:31.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:30.977+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:31.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:30.977+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:31.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:30.977+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:31.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:31.448+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:31.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:31.448+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:31.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:31.448+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:31.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:31.449+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:31.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:31.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:31.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:31.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:31.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:31.561+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:31.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:31.562+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:31.983+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:31.983+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:31.983+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:32.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:31.984+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:32.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:32:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:32:32.365+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45156 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:32:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:32.419+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:32.419+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:32.419+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:32.420+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:32.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:32.581+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:32.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:32.581+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:32.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:32.581+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:32.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:32.582+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:33.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:33.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:33.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:33.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:33.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:33.016+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:33.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:33.016+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:33.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:33.409+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:33.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:33.409+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:33.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:33.409+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:33.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:33.410+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:33.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:33.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:33.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:33.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:33.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:33.574+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:33.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:33.574+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:34.447 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:34.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:34.447 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:34.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:34.447 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:34.006+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:34.447 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:34.006+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:34.444+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:34.444+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:34.444+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:34.444+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:34.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:34.613+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:34.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:34.613+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:34.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:34.613+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:34.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:34.614+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:35.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:35.044+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:35.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:35.044+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:35.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:35.044+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:35.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:35.044+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:35.565+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:35.565+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:35.565+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:35.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:35.565+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:35.421+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:35.421+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:35.421+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:35.421+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:36.401 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:36.032+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:36.401 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:36.032+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:36.401 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:36.032+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:36.401 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:36.033+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:36.398+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:36.398+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:36.398+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:36.399+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:36.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:36.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:36.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:36.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:36.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:36.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:36.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:36.596+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:37.027+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:37.027+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:37.027+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:37.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:37.028+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:37.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:32:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:32:37.372+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45202 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:32:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:37.421+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:37.421+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:37.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:37.421+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:37.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:37.422+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:37.587+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:37.587+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:37.587+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:37.588+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:38.068+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:38.068+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:38.068+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:38.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:38.069+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:38.414+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:38.414+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:38.414+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:38.414+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:38.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:38.602+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:38.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:38.602+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:38.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:38.602+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:38.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:38.603+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:39.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:39.095+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:39.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:39.095+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:39.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:39.095+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:39.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:39.096+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:39.427+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:39.427+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:39.427+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:39.428+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:39.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:39.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:39.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:39.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:39.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:39.596+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:39.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:39.597+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:40.385 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:40.068+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:40.385 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:40.068+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:40.386 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:40.068+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:40.386 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:40.069+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:40.383+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:40.383+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:40.383+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:40.383+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:40.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:40.551+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:40.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:40.551+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:40.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:40.551+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:40.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:40.552+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:41.402 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:41.075+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:41.403 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:41.075+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:41.403 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:41.075+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:41.403 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:41.076+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:41.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:41.400+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:41.400+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:41.400+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:41.400+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:41.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:41.570+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:41.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:41.570+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:41.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:41.570+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:41.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:41.571+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:42.360 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:42.096+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:42.360 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:42.096+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:42.360 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:42.096+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:42.360 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:42.097+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:42.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:32:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:32:42.378+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45246 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:32:42.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:42.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:42.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:42.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:42.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:42.580+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:42.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:42.580+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:42.357+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:42.357+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:42.357+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:42.358+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:43.405 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:43.083+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:43.406 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:43.083+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:43.406 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:43.083+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:43.406 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:43.083+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:43.406 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:43.403+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:43.406 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:43.403+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:43.406 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:43.403+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:43.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:43.404+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:43.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:43.579+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:43.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:43.579+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:43.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:43.579+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:43.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:43.580+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:44.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:44.114+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:44.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:44.114+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:44.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:44.114+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:44.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:44.115+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:44.587+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:44.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:44.587+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:44.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:44.587+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:44.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:44.587+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:44.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:44.426+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:44.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:44.426+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:44.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:44.426+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:44.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:44.427+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:45.152+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:45.152+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:45.152+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:45.153+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:45.390+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:45.390+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:45.390+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:45.391+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:45.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:45.548+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:45.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:45.548+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:45.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:45.548+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:45.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:45.549+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:46.392 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:46.107+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:46.392 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:46.107+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:46.392 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:46.107+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:46.392 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:46.107+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:46.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:46.539+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:46.539+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:46.539+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:46.539+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:46.390+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:46.390+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:46.390+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:46.390+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:47.386 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:47.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:47.387 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:47.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:47.387 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:47.067+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:47.387 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:47.067+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:47.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:32:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:32:47.384+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45293 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:32:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:47.426+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:47.426+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:47.426+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:47.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:47.426+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:47.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:47.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:47.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:47.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:47.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:47.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:47.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:47.589+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:48.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:48.112+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:48.112+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:48.112+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:48.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:48.112+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:48.465+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:48.465+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:48.465+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:48.465+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:48.589+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:48.589+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:48.589+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:48.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:48.590+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:49.106+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:49.106+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:49.106+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:49.107+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:49.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:49.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:49.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:49.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:49.491+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:49.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:49.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:49.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:49.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:49.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:49.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:49.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:49.589+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:50.083+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:50.083+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:50.083+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:50.084+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:50.537+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:50.537+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:50.537+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:50.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:50.538+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:50.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:50.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:50.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:50.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:50.600+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:50.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:50.601+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:51.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:51.073+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:51.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:51.073+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:51.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:51.073+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:51.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:51.074+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:51.570+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:51.570+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:51.570+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:51.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:51.571+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:51.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:51.554+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:51.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:51.554+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:51.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:51.554+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:51.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:51.555+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:52.368 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:52.070+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:52.368 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:52.070+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:52.369 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:52.070+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:52.369 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:52.070+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:52.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:32:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:32:52.390+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45337 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:32:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:52.570+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:52.570+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:52.570+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:52.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:52.570+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:52.529+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:52.529+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:52.529+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:52.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:52.529+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:53.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:53.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:53.052+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:53.053+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:53.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:53.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:53.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:53.577+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:53.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:53.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:53.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:53.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:53.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:53.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:53.574+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:54.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:54.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:54.063+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:54.063+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:54.599+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:54.599+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:54.599+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:54.599+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:54.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:54.546+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:54.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:54.546+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:54.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:54.546+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:54.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:54.547+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:55.440 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:55.100+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:55.441 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:55.100+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:55.441 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:55.100+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:55.441 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:55.101+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:55.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:55.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:55.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:55.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:55.504+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:55.556+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:55.556+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:55.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:55.556+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:55.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:55.557+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:56.072+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:56.072+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:56.072+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:56.073+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:56.562+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:56.562+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:56.562+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:56.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:56.563+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:56.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:56.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:56.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:56.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:56.479+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:56.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:56.480+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:57.398 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:57.104+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:57.399 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:57.104+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:57.399 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:57.104+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:57.399 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:57.105+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:57.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:32:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:32:57.396+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45383 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:32:57.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:57.547+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:57.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:57.547+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:57.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:57.547+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:57.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:57.548+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:57.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:57.521+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:57.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:57.521+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:57.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:57.521+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:57.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:57.522+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:58.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:58.097+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:58.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:58.097+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:58.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:58.097+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:58.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:58.098+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:58.501+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:58.501+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:58.501+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:58.502+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:58.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:58.517+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:58.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:58.517+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:58.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:58.517+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:58.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:58.518+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:59.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:59.080+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:59.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:59.080+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:59.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:59.080+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:32:59.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:32:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:32:59.081+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:32:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:59.498+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:59.498+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:59.498+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:32:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:32:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:32:59.498+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:32:59.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:59.477+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:59.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:59.477+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:59.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:59.477+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:32:59.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:32:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:32:59.477+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:00.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:00.057+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:00.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:00.057+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:00.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:00.057+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:00.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:00.058+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:00.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:00.467+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:00.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:00.467+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:00.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:00.467+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:00.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:00.468+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:00.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:00.540+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:00.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:00.540+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:00.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:00.540+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:00.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:00.541+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:01.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:01.081+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:01.081+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:01.081+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:01.081+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:01.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:01.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:01.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:01.482+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:01.521+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:01.522+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:01.522+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:01.522+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:02.404 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:02.100+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:02.405 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:02.100+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:02.405 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:02.100+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:02.405 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:02.100+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:02.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:33:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:33:02.402+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45427 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:33:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:02.471+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:02.471+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:02.471+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:02.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:02.472+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:02.560+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:02.560+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:02.560+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:02.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:02.560+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:03.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:03.070+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:03.070+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:03.070+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:03.071+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:03.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:03.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:03.586+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:03.586+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:03.465+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:03.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:03.465+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:03.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:03.465+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:03.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:03.466+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:04.100+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:04.100+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:04.100+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:04.101+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:04.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:04.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:04.466+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:04.467+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:04.628+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:04.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:04.628+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:04.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:04.628+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:04.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:04.629+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:05.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:05.117+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:05.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:05.117+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:05.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:05.117+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:05.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:05.117+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:05.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:05.446+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:05.447+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:05.447+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:05.447+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:05.616+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:05.616+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:05.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:05.616+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:05.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:05.617+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:06.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:06.154+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:06.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:06.154+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:06.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:06.154+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:06.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:06.155+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:06.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:06.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:06.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:06.666+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:06.480+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:06.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:06.480+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:06.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:06.480+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:06.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:06.481+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:07.410 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:07.129+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:07.410 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:07.129+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:07.410 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:07.129+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:07.411 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:07.130+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:07.702 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:33:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:33:07.408+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45474 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:33:07.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:07.451+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:07.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:07.451+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:07.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:07.451+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:07.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:07.452+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:07.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:07.699+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:07.699+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:07.699+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:07.700+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:08.171+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:08.171+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:08.171+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:08.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:08.172+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:08.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:08.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:08.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:08.437+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:08.437+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:08.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:08.655+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:08.655+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:08.655+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:08.656+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:09.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:09.168+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:09.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:09.168+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:09.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:09.168+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:09.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:09.169+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:09.658+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:09.658+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:09.658+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:09.659+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:09.477+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:09.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:09.477+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:09.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:09.477+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:09.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:09.478+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:10.514 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:10.218+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:10.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:10.218+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:10.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:10.218+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:10.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:10.219+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:10.801 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:10.512+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:10.802 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:10.512+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:10.802 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:10.512+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:10.802 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:10.513+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:10.802 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:10.707+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:10.802 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:10.707+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:10.802 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:10.707+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:10.802 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:10.708+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:11.192+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:11.192+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:11.192+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:11.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:11.193+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:11.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:11.548+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:11.548+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:11.548+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:11.549+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:11.680+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:11.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:11.680+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:11.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:11.680+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:11.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:11.680+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:12.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:33:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:33:12.414+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45518 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:33:12.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:12.230+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:12.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:12.230+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:12.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:12.230+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:12.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:12.231+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:12.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:12.546+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:12.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:12.546+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:12.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:12.546+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:12.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:12.547+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:12.859 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:12.711+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:12.859 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:12.711+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:12.859 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:12.711+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:12.859 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:12.712+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:13.544 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:13.242+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:13.544 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:13.242+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:13.545 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:13.242+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:13.545 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:13.242+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:13.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:13.542+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:13.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:13.542+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:13.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:13.542+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:13.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:13.543+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:13.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:13.723+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:13.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:13.723+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:13.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:13.723+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:13.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:13.724+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:14.552 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:14.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:14.552 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:14.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:14.552 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:14.276+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:14.553 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:14.277+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:14.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:14.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:14.550+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:14.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:14.550+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:14.771+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:14.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:14.771+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:14.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:14.771+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:14.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:14.772+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:15.543 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:15.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:15.543 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:15.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:15.543 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:15.275+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:15.543 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:15.276+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:15.540+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:15.540+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:15.540+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:15.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:15.541+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:15.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:15.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:15.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:15.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:15.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:15.768+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:16.318+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:16.318+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:16.318+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:16.319+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:16.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:16.558+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:16.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:16.558+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:16.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:16.558+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:16.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:16.558+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:16.769+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:16.769+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:16.769+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:16.769+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:17.515 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:33:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:33:17.419+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45564 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:33:17.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:17.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:17.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:17.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:17.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:17.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:17.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:17.309+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:17.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:17.512+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:17.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:17.512+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:17.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:17.512+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:17.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:17.513+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:18.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:17.775+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:17.775+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:17.775+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:17.776+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:18.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:18.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:18.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:18.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:18.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:18.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:18.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:18.343+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:18.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:18.553+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:18.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:18.553+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:18.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:18.553+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:18.719 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:18.554+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:18.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:18.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:18.811+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:18.812+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:19.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:19.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:19.386+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:19.386+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:19.545+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:19.545+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:19.545+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:19.546+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:19.763+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:19.763+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:19.763+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:19.764+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:20.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:20.430+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:20.430+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:20.430+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:20.430+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:20.496+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:20.496+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:20.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:20.496+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:20.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:20.497+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:20.741+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:20.741+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:20.741+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:20.742+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:21.751 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:21.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:21.751 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:21.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:21.751 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:21.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:21.751 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:21.465+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:21.751 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:21.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:21.752 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:21.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:21.752 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:21.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:21.752 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:21.487+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:21.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:21.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:21.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:21.749+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:22.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:33:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:33:22.425+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45608 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:33:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:22.480+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:22.480+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:22.480+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:22.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:22.481+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:22.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:22.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:22.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:22.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:22.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:22.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:22.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:22.491+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:23.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:22.778+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:23.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:22.778+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:23.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:22.778+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:23.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:22.779+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:23.789 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:23.496+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:23.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:23.496+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:23.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:23.496+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:23.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:23.497+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:23.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:23.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:23.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:23.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:23.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:23.468+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:23.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:23.469+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:24.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:23.787+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:23.787+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:23.787+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:23.787+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:24.451+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:24.451+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:24.451+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:24.452+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:24.477+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:24.477+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:24.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:24.477+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:24.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:24.478+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:25.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:24.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:24.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:24.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:24.832+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:25.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:25.481+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:25.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:25.481+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:25.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:25.481+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:25.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:25.482+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:25.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:25.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:25.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:25.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:25.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:25.523+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:25.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:25.524+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:26.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:25.789+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:26.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:25.789+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:26.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:25.789+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:26.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:25.790+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:26.827 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:26.470+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:26.827 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:26.470+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:26.827 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:26.470+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:26.827 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:26.470+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:26.828 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:26.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:26.828 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:26.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:26.828 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:26.486+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:26.828 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:26.487+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:27.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:26.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:26.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:26.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:27.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:26.825+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:33:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:33:27.431+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45655 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:33:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:27.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:27.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:27.506+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:27.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:27.507+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:27.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:27.500+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:27.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:27.500+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:27.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:27.500+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:27.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:27.500+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:27.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:27.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:27.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:27.784+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:28.823 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:28.544+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:28.823 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:28.544+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:28.823 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:28.544+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:28.823 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:28.544+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:28.824 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:28.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:28.824 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:28.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:28.824 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:28.481+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:28.824 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:28.482+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:29.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:28.820+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:28.820+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:28.820+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:28.821+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:29.540+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:29.540+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:29.540+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:29.541+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:29.440+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:29.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:29.440+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:29.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:29.440+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:29.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:29.441+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:30.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:29.793+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:29.793+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:29.793+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:29.793+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:30.510+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:30.510+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:30.510+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:30.510+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:30.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:30.455+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:30.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:30.455+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:30.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:30.455+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:30.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:30.456+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:30.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:30.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:30.752+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:30.753+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:31.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:31.420+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:31.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:31.420+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:31.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:31.420+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:31.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:31.420+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:31.504+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:31.504+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:31.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:31.504+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:31.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:31.504+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:32.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:31.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:31.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:31.791+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:31.792+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:32.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:33:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:33:32.437+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45699 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:33:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:32.492+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:32.492+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:32.492+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:32.493+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:32.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:32.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:32.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:32.461+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:33.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:32.779+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:33.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:32.779+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:33.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:32.779+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:33.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:32.780+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:33.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:33.513+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:33.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:33.513+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:33.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:33.513+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:33.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:33.513+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:33.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:33.450+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:33.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:33.450+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:33.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:33.450+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:33.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:33.451+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:33.808+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:33.808+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:33.808+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:34.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:33.809+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:34.485+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:34.485+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:34.485+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:34.486+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:34.418+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:34.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:34.418+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:34.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:34.418+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:34.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:34.418+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:34.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:34.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:34.819+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:35.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:34.820+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:35.474+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:35.474+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:35.474+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:35.475+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:35.388+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:35.388+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:35.388+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:35.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:35.389+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:35.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:35.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:35.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:36.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:35.865+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:36.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:36.465+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:36.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:36.465+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:36.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:36.465+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:36.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:36.466+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:36.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:36.407+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:36.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:36.407+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:36.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:36.407+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:36.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:36.408+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:36.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:36.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:36.831+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:37.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:36.832+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:37.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:33:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:33:37.443+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45745 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:33:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:37.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:37.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:37.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:37.441+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:37.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:37.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:37.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:37.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:37.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:37.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:37.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:37.366+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:37.882+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:37.882+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:37.882+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:37.883+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:38.363+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:38.363+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:38.363+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:38.364+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:38.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:38.393+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:38.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:38.393+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:38.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:38.393+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:38.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:38.394+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:38.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:38.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:38.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:39.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:38.919+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:39.404+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:39.404+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:39.404+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:39.405+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:39.412+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:39.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:39.412+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:39.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:39.412+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:39.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:39.412+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:40.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:39.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:39.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:39.900+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:39.900+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:40.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:40.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:40.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:40.374+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:40.371+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:40.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:40.371+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:40.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:40.371+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:40.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:40.372+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:40.917+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:40.917+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:40.917+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:41.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:40.917+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:41.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:41.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:41.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:41.383+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:41.352+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:41.352+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:41.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:41.352+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:41.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:41.353+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:41.878+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:41.878+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:41.878+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:41.879+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:42.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:33:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:33:42.449+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45789 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:33:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:42.388+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:42.388+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:42.388+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:42.388+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:42.320+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:42.320+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:42.320+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:42.320+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:43.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:42.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:43.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:42.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:43.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:42.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:43.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:42.919+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:43.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:43.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:43.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:43.387+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:43.278+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:43.278+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:43.278+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:43.279+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:43.940+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:43.940+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:43.940+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:43.941+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:44.592 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:44.353+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:44.592 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:44.353+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:44.593 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:44.353+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:44.593 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:44.354+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:44.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:44.262+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:44.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:44.262+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:44.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:44.262+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:44.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:44.263+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:44.902+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:44.902+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:44.902+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:45.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:44.903+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:45.377+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:45.377+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:45.377+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:45.378+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:45.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:45.257+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:45.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:45.257+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:45.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:45.257+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:45.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:45.258+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:45.896+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:45.896+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:45.896+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:45.896+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:46.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:46.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:46.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:46.382+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:46.287+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:46.287+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:46.287+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:46.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:46.288+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:46.847+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:46.847+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:46.847+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:47.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:46.848+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:47.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:33:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:33:47.455+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45836 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:33:47.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:47.377+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:47.377+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:47.377+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:47.378+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:47.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:47.296+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:47.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:47.296+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:47.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:47.296+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:47.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:47.297+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:47.883+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:47.883+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:47.883+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:48.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:47.884+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:48.411+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:48.411+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:48.411+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:48.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:48.412+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:48.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:48.274+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:48.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:48.274+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:48.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:48.274+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:48.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:48.274+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:48.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:48.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:48.872+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:49.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:48.873+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:49.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:49.426+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:49.426+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:49.426+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:49.427+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:49.287+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:49.287+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:49.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:49.287+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:49.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:49.288+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:50.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:49.905+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:50.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:49.905+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:50.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:49.905+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:50.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:49.906+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:50.273+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:50.273+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:50.273+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:50.273+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:50.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:50.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:50.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:50.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:50.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:50.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:50.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:50.402+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:50.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:50.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:50.932+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:50.933+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:51.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:51.353+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:51.353+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:51.353+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:51.354+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:51.318+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:51.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:51.318+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:51.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:51.318+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:51.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:51.319+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:52.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:51.945+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:51.946+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:51.946+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:52.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:51.946+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:52.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:33:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:33:52.461+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45880 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:33:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:52.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:52.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:52.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:52.332+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:52.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:52.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:52.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:52.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:52.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:52.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:52.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:52.284+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:52.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:52.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:52.953+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:53.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:52.954+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:53.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:53.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:53.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:53.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:53.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:53.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:53.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:53.330+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:53.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:53.328+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:53.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:53.328+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:53.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:53.328+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:53.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:53.328+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:54.304 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:53.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:54.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:53.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:54.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:53.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:54.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:53.981+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:54.347+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:54.347+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:54.347+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:54.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:54.348+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:54.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:54.302+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:54.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:54.302+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:54.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:54.302+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:54.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:54.302+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:55.294 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:54.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:55.294 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:54.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:55.294 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:54.980+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:55.294 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:54.981+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:55.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:55.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:55.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:55.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:55.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:55.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:55.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:55.334+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:55.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:55.291+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:55.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:55.291+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:55.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:55.291+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:55.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:55.292+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:56.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:55.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:56.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:55.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:56.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:55.979+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:56.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:55.980+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:56.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:56.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:56.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:56.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:56.383+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:56.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:56.261+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:56.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:56.261+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:56.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:56.261+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:56.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:56.262+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:57.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:56.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:57.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:56.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:57.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:56.981+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:57.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:56.982+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:57.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:33:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:33:57.467+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45926 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:33:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:57.407+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:57.407+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:57.407+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:57.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:57.408+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:57.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:57.269+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:57.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:57.270+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:57.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:57.270+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:57.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:57.270+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:58.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:57.945+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:57.945+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:57.945+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:58.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:57.946+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:58.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:58.308+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:58.308+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:58.308+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:58.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:58.309+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:58.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:58.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:58.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:58.392+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:58.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:58.392+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:33:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:58.913+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:58.913+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:58.913+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:33:59.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:58.914+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:59.357+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:59.357+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:59.357+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:33:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:33:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:33:59.357+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:33:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:59.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:59.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:59.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:59.416+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:33:59.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:33:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:33:59.416+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:59.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:59.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:59.924+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:00.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:33:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:33:59.925+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:00.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:00.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:00.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:00.389+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:00.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:00.390+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:00.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:00.367+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:00.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:00.367+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:00.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:00.367+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:00.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:00.368+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:00.906+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:00.906+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:00.906+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:01.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:00.907+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:01.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:01.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:01.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:01.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:01.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:01.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:01.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:01.356+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:01.344+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:01.344+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:01.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:01.344+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:01.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:01.345+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:02.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:01.893+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:01.893+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:01.893+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:02.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:01.894+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:02.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:34:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:34:02.473+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 45970 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:34:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:02.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:02.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:02.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:02.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:02.340+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:02.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:02.325+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:02.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:02.325+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:02.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:02.325+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:02.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:02.326+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:03.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:02.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:03.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:02.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:03.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:02.904+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:03.026 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:02.905+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:03.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:03.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:03.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:03.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:03.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:03.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:03.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:03.329+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:03.549 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:03.294+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:03.550 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:03.294+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:03.550 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:03.294+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:03.550 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:03.295+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:03.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:03.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:03.918+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:04.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:03.919+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:04.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:04.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:04.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:04.335+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:04.335+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:04.281+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:04.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:04.281+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:04.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:04.281+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:04.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:04.282+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:05.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:04.925+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:04.925+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:04.925+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:05.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:04.926+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:05.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:05.313+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:05.313+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:05.313+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:05.314+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:05.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:05.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:05.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:05.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:05.307+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:05.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:05.308+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:06.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:05.938+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:05.938+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:05.938+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:06.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:05.938+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:06.281+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:06.281+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:06.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:06.281+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:06.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:06.282+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:06.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:06.352+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:06.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:06.352+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:06.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:06.352+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:06.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:06.353+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:07.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:06.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:06.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:06.909+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:07.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:06.910+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:07.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:34:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:34:07.479+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46017 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:34:07.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:07.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:07.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:07.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:07.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:07.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:07.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:07.329+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:07.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:07.375+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:07.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:07.375+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:07.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:07.375+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:07.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:07.375+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:07.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:07.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:07.891+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:08.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:07.892+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:08.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:08.313+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:08.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:08.313+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:08.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:08.313+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:08.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:08.313+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:08.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:08.378+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:08.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:08.378+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:08.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:08.378+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:08.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:08.378+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:08.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:08.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:08.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:08.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:08.868+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:09.357+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:09.357+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:09.357+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:09.358+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:09.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:09.362+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:09.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:09.362+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:09.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:09.362+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:09.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:09.363+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:10.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:09.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:09.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:09.889+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:10.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:09.890+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:10.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:10.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:10.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:10.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:10.378+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:10.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:10.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:10.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:10.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:10.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:10.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:10.322+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:10.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:10.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:10.890+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:11.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:10.891+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:11.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:11.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:11.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:11.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:11.333+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:11.303+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:11.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:11.303+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:11.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:11.303+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:11.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:11.304+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:12.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:11.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:11.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:11.869+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:11.869+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:12.496 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:34:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:34:12.485+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46061 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:34:12.496 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:12.303+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:12.496 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:12.303+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:12.497 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:12.303+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:12.497 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:12.304+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:12.497 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:12.284+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:12.497 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:12.284+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:12.497 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:12.284+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:12.497 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:12.284+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:13.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:12.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:12.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:12.867+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:13.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:12.868+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:13.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:13.295+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:13.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:13.295+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:13.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:13.295+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:13.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:13.296+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:13.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:13.322+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:13.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:13.322+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:13.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:13.322+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:13.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:13.323+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:13.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:13.861+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:13.861+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:13.861+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:13.862+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:14.256+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:14.256+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:14.256+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:14.257+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:14.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:14.276+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:14.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:14.276+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:14.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:14.276+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:14.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:14.277+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:14.861+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:14.861+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:14.861+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:15.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:14.862+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:15.281+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:15.281+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:15.281+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:15.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:15.282+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:15.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:15.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:15.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:15.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:15.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:15.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:15.322+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:16.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:15.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:15.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:15.865+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:16.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:15.865+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:16.252+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:16.252+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:16.252+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:16.253+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:16.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:16.353+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:16.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:16.353+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:16.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:16.353+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:16.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:16.354+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:16.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:16.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:17.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:16.824+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:17.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:16.825+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:17.494 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:17.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:17.494 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:17.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:17.494 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:17.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:17.495 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:17.285+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:17.495 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:17.333+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:17.495 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:17.333+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:17.495 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:17.333+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:17.495 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:17.333+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:17.846 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:34:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:34:17.491+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46107 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:34:18.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:17.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:17.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:17.844+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:18.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:17.844+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:18.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:18.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:18.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:18.279+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:18.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:18.313+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:18.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:18.313+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:18.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:18.313+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:18.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:18.313+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:18.838+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:18.838+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:18.838+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:19.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:18.839+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:19.318+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:19.318+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:19.318+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:19.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:19.319+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:19.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:19.246+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:19.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:19.246+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:19.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:19.246+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:19.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:19.247+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:19.878+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:19.878+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:19.878+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:19.878+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:20.347+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:20.347+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:20.347+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:20.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:20.348+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:20.280+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:20.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:20.280+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:20.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:20.280+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:20.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:20.281+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:20.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:20.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:20.928+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:20.929+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:21.319+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:21.319+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:21.319+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:21.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:21.320+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:21.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:21.245+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:21.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:21.245+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:21.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:21.245+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:21.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:21.246+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:21.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:21.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:21.933+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:22.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:21.933+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:22.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:34:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:34:22.498+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46151 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:34:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:22.264+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:22.264+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:22.264+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:22.265+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:22.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:22.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:22.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:22.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:22.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:22.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:22.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:22.366+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:22.954+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:22.954+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:22.954+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:23.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:22.954+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:23.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:23.272+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:23.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:23.272+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:23.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:23.272+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:23.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:23.273+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:23.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:23.351+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:23.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:23.351+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:23.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:23.351+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:23.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:23.352+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:23.950+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:23.950+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:23.950+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:24.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:23.951+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:24.528 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:24.246+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:24.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:24.246+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:24.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:24.246+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:24.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:24.246+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:24.530 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:24.319+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:24.530 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:24.319+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:24.530 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:24.319+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:24.531 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:24.320+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:25.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:24.962+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:24.962+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:24.962+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:24.963+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:25.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:25.278+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:25.278+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:25.278+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:25.279+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:25.368+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:25.368+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:25.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:25.368+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:25.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:25.368+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:25.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:25.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:25.921+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:25.921+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:26.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:26.402+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:26.402+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:26.402+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:26.403+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:26.317+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:26.317+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:26.317+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:26.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:26.318+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:27.286 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:26.970+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:27.287 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:26.970+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:27.287 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:26.970+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:27.287 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:26.971+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:27.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:34:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:34:27.504+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46195 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:34:27.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:27.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:27.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:27.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:27.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:27.284+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:27.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:27.285+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:27.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:27.452+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:27.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:27.452+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:27.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:27.452+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:27.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:27.453+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:28.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:27.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:27.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:27.934+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:27.935+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:28.404+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:28.404+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:28.404+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:28.404+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:28.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:28.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:28.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:28.334+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:28.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:28.334+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:28.910+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:28.911+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:28.911+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:28.911+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:29.339+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:29.339+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:29.339+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:29.340+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:29.410+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:29.410+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:29.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:29.410+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:29.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:29.411+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:29.940+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:29.940+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:29.940+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:30.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:29.940+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:30.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:30.422+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:30.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:30.422+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:30.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:30.422+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:30.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:30.423+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:30.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:30.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:30.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:30.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:30.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:30.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:30.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:30.344+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:31.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:30.961+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:30.961+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:30.961+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:31.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:30.962+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:31.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:31.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:31.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:31.379+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:31.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:31.418+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:31.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:31.418+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:31.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:31.418+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:31.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:31.419+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:32.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:32.007+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:32.351 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:32.007+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:32.351 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:32.007+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:32.351 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:32.008+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:32.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:34:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:34:32.510+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46239 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:34:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:32.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:32.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:32.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:32.348+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:32.456+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:32.456+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:32.456+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:32.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:32.456+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:33.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:33.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:33.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:33.009+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:33.009+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:33.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:33.383+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:33.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:33.383+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:33.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:33.383+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:33.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:33.384+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:33.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:33.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:33.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:33.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:33.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:33.505+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:33.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:33.506+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:34.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:34.054+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:34.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:34.054+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:34.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:34.054+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:34.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:34.054+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:34.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:34.368+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:34.368+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:34.368+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:34.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:34.369+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:34.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:34.510+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:34.511+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:34.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:34.511+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:35.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:35.022+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:35.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:35.022+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:35.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:35.022+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:35.333 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:35.023+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:35.471+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:35.471+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:35.471+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:35.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:35.472+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:35.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:35.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:35.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:35.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:35.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:35.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:35.330+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:36.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:36.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:36.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:36.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:36.059+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:36.060+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:36.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:36.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:36.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:36.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:36.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:36.294+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:36.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:36.295+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:36.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:36.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:36.514+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:36.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:36.514+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:37.330 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:37.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:37.331 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:37.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:37.331 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:37.070+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:37.331 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:37.071+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:37.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:34:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:34:37.515+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46285 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:34:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:37.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:37.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:37.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:37.328+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:37.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:37.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:37.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:37.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:37.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:37.522+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:37.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:37.523+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:38.053+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:38.053+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:38.053+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:38.054+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:38.323+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:38.323+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:38.323+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:38.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:38.323+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:38.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:38.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:38.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:38.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:38.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:38.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:38.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:38.475+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:39.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:39.045+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:39.346 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:39.045+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:39.346 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:39.045+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:39.346 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:39.046+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:39.448+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:39.448+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:39.448+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:39.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:39.449+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:39.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:39.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:39.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:39.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:39.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:39.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:39.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:39.344+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:40.288 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:40.020+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:40.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:40.020+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:40.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:40.020+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:40.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:40.021+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:40.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:40.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:40.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:40.343+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:40.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:40.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:40.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:40.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:40.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:40.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:40.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:40.424+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:41.343 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:41.061+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:41.343 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:41.061+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:41.343 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:41.061+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:41.343 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:41.062+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:41.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:41.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:41.340+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:41.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:41.341+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:41.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:41.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:41.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:41.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:41.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:41.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:41.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:41.424+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:42.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:42.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:42.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:42.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:42.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:42.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:42.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:42.299+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:42.422 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:42.099+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:42.422 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:42.099+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:42.422 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:42.099+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:42.422 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:42.100+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:42.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:34:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:34:42.521+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46329 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:34:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:42.418+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:42.418+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:42.418+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:42.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:42.419+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:43.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:43.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:43.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:43.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:43.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:43.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:43.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:43.329+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:43.426 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:43.124+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:43.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:43.124+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:43.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:43.124+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:43.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:43.125+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:43.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:43.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:43.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:43.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:43.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:43.423+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:43.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:43.424+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:44.454 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:44.374+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:44.454 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:44.374+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:44.454 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:44.374+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:44.454 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:44.375+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:44.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:44.129+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:44.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:44.129+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:44.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:44.129+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:44.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:44.129+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:44.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:44.451+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:44.451+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:44.451+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:44.452+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:45.359+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:45.359+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:45.359+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:45.359+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:45.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:45.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:45.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:45.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:45.141+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:45.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:45.142+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:45.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:45.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:45.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:45.482+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:45.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:45.482+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:46.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:46.163+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:46.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:46.163+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:46.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:46.163+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:46.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:46.164+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:46.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:46.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:46.460+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:46.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:46.460+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:46.357+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:46.357+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:46.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:46.357+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:46.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:46.357+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:47.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:47.162+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:47.162+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:47.162+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:47.162+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:47.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:47.376+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:47.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:47.376+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:47.376+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:47.377+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:47.871 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:34:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:34:47.527+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46376 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:34:47.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:47.502+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:47.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:47.502+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:47.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:47.502+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:47.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:47.503+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:48.207+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:48.207+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:48.207+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:48.208+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:48.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:48.419+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:48.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:48.419+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:48.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:48.419+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:48.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:48.420+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:48.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:48.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:48.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:48.475+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:48.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:48.476+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:49.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:49.161+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:49.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:49.161+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:49.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:49.161+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:49.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:49.162+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:49.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:49.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:49.525+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:49.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:49.526+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:49.429+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:49.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:49.429+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:49.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:49.429+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:49.807 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:49.430+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:50.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:50.161+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:50.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:50.161+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:50.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:50.161+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:50.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:50.162+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:50.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:50.434+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:50.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:50.434+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:50.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:50.434+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:50.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:50.435+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:50.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:50.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:50.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:50.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:50.573+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:50.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:50.573+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:51.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:51.154+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:51.416 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:51.154+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:51.416 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:51.154+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:51.416 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:51.155+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:51.549+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:51.549+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:51.549+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:51.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:51.550+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:51.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:51.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:51.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:51.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:51.412+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:51.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:51.413+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:52.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:52.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:52.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:52.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:52.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:52.109+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:52.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:52.110+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:52.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:34:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:34:52.532+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46420 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:34:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:52.379+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:52.379+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:52.379+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:52.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:52.380+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:52.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:52.512+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:52.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:52.512+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:52.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:52.512+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:52.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:52.512+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:53.336+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:53.336+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:53.336+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:53.337+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:53.107+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:53.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:53.107+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:53.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:53.107+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:53.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:53.109+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:53.559+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:53.559+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:53.559+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:53.560+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:54.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:54.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:54.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:54.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:54.346+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:54.128+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:54.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:54.128+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:54.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:54.128+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:54.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:54.129+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:54.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:54.527+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:54.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:54.527+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:54.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:54.527+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:54.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:54.528+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:55.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:55.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:55.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:55.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:55.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:55.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:55.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:55.333+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:55.177+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:55.177+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:55.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:55.177+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:55.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:55.178+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:55.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:55.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:55.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:55.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:55.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:55.503+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:55.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:55.503+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:56.326+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:56.326+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:56.326+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:56.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:56.327+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:56.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:56.165+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:56.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:56.165+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:56.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:56.165+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:56.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:56.166+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:56.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:56.552+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:56.552+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:56.552+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:56.553+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:57.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:57.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:57.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:57.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:57.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:57.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:57.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:57.298+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:57.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:57.176+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:57.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:57.176+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:57.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:57.176+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:57.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:57.177+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:57.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:34:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:34:57.539+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46466 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:34:57.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:57.521+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:57.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:57.521+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:57.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:57.521+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:57.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:57.521+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:58.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:58.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:58.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:58.279+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:58.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:58.280+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:58.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:58.203+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:58.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:58.203+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:58.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:58.203+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:58.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:58.204+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:58.518+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:58.518+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:58.518+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:58.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:58.519+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:59.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:59.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:59.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:59.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:59.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:59.490+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:34:59.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:34:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:34:59.490+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:34:59.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:59.290+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:59.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:59.290+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:59.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:59.290+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:34:59.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:34:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:34:59.291+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:34:59.599 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:59.242+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:59.599 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:59.242+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:59.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:59.242+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:34:59.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:34:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:34:59.242+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:00.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:00.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:00.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:00.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:00.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:00.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:00.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:00.333+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:00.537 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:00.221+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:00.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:00.221+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:00.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:00.221+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:00.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:00.221+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:00.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:00.534+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:00.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:00.534+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:00.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:00.534+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:00.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:00.535+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:01.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:01.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:01.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:01.374+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:01.175+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:01.175+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:01.175+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:01.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:01.175+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:01.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:01.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:01.578+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:01.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:01.578+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:02.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:02.347+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:02.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:02.347+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:02.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:02.347+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:02.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:02.347+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:02.212+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:02.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:02.212+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:02.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:02.212+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:02.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:02.213+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:02.860 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:35:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:35:02.545+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46510 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:35:02.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:02.624+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:02.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:02.624+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:02.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:02.624+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:02.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:02.624+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:03.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:03.200+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:03.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:03.200+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:03.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:03.200+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:03.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:03.201+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:03.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:03.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:03.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:03.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:03.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:03.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:03.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:03.331+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:03.642+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:03.642+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:03.642+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:03.643+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:04.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:04.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:04.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:04.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:04.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:04.351+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:04.615 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:04.352+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:04.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:04.221+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:04.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:04.221+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:04.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:04.221+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:04.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:04.222+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:04.611+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:04.611+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:04.611+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:04.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:04.612+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:05.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:05.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:05.619 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:05.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:05.619 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:05.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:05.619 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:05.383+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:05.619 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:05.256+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:05.619 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:05.256+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:05.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:05.256+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:05.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:05.256+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:05.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:05.616+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:05.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:05.616+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:05.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:05.616+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:05.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:05.617+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:06.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:06.431+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:06.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:06.431+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:06.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:06.431+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:06.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:06.432+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:06.655 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:06.230+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:06.655 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:06.230+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:06.655 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:06.230+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:06.655 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:06.231+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:06.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:06.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:06.651+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:06.652+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:06.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:06.653+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:07.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:07.199+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:07.199+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:07.199+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:07.200+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:07.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:35:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:35:07.551+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46557 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:35:07.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:07.681+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:07.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:07.681+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:07.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:07.681+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:07.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:07.682+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:07.876 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:07.470+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:07.876 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:07.470+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:07.876 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:07.470+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:07.876 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:07.471+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:08.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:08.189+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:08.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:08.189+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:08.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:08.189+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:08.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:08.189+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:08.454+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:08.455+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:08.455+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:08.455+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:08.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:08.642+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:08.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:08.642+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:08.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:08.642+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:08.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:08.643+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:09.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:09.424+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:09.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:09.424+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:09.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:09.424+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:09.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:09.424+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:09.619 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:09.217+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:09.619 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:09.217+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:09.619 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:09.217+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:09.619 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:09.218+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:09.615+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:09.615+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:09.615+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:09.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:09.616+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:10.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:10.189+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:10.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:10.189+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:10.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:10.189+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:10.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:10.189+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:10.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:10.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:10.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:10.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:10.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:10.464+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:10.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:10.465+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:10.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:10.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:10.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:10.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:10.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:10.620+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:10.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:10.620+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:11.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:11.144+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:11.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:11.144+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:11.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:11.144+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:11.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:11.145+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:11.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:11.484+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:11.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:11.484+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:11.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:11.484+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:11.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:11.485+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:11.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:11.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:11.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:11.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:11.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:11.644+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:11.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:11.645+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:12.442 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:12.169+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:12.442 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:12.169+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:12.442 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:12.169+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:12.442 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:12.170+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:12.716 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:35:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:35:12.557+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46601 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:35:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:12.438+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:12.438+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:12.438+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:12.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:12.438+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:12.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:12.648+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:12.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:12.648+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:12.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:12.648+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:12.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:12.648+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:13.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:13.204+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:13.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:13.204+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:13.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:13.204+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:13.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:13.205+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:13.696+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:13.696+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:13.696+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:13.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:13.697+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:13.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:13.487+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:13.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:13.487+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:13.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:13.487+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:13.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:13.488+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:14.536 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:14.224+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:14.536 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:14.224+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:14.537 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:14.224+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:14.537 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:14.225+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:14.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:14.534+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:14.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:14.534+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:14.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:14.534+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:14.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:14.534+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:14.832 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:14.670+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:14.832 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:14.670+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:14.832 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:14.670+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:14.832 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:14.671+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:15.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:15.197+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:15.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:15.197+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:15.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:15.197+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:15.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:15.198+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:15.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:15.581+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:15.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:15.581+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:15.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:15.581+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:15.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:15.582+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:15.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:15.633+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:15.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:15.633+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:15.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:15.633+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:15.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:15.633+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:16.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:16.160+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:16.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:16.160+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:16.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:16.160+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:16.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:16.161+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:16.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:16.591+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:16.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:16.591+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:16.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:16.591+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:16.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:16.591+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:16.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:16.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:16.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:16.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:16.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:16.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:16.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:16.677+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:17.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:17.156+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:17.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:17.156+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:17.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:17.156+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:17.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:17.157+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:17.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:35:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:35:17.563+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46647 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:35:17.877 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:17.591+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:17.877 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:17.591+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:17.877 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:17.591+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:17.877 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:17.591+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:17.878 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:17.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:17.878 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:17.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:17.878 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:17.649+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:17.878 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:17.649+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:18.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:18.192+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:18.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:18.192+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:18.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:18.192+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:18.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:18.193+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:18.580+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:18.580+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:18.580+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:18.581+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:18.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:18.667+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:18.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:18.667+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:18.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:18.667+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:18.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:18.668+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:19.545 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:19.217+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:19.545 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:19.217+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:19.545 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:19.217+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:19.545 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:19.218+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:19.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:19.543+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:19.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:19.543+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:19.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:19.543+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:19.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:19.543+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:19.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:19.710+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:19.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:19.710+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:19.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:19.710+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:19.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:19.710+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:20.516 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:20.243+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:20.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:20.243+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:20.590 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:20.243+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:20.590 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:20.243+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:20.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:20.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:20.732+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:20.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:20.733+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:20.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:20.514+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:20.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:20.514+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:20.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:20.514+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:20.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:20.515+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:21.563 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:21.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:21.563 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:21.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:21.564 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:21.229+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:21.564 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:21.230+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:21.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:21.685+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:21.685+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:21.685+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:21.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:21.686+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:21.561+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:21.561+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:21.561+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:21.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:21.562+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:22.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:22.215+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:22.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:22.215+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:22.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:22.215+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:22.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:22.216+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:22.765 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:35:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:35:22.569+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46691 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:35:22.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:22.512+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:22.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:22.512+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:22.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:22.512+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:22.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:22.513+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:22.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:22.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:22.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:22.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:22.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:22.676+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:22.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:22.677+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:23.190+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:23.190+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:23.190+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:23.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:23.191+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:23.463+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:23.463+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:23.463+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:23.464+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:23.704+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:23.704+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:23.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:23.704+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:23.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:23.705+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:24.419+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:24.419+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:24.419+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:24.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:24.420+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:24.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:24.181+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:24.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:24.181+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:24.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:24.181+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:24.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:24.182+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:25.044 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:24.753+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:25.044 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:24.753+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:25.045 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:24.753+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:25.045 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:24.753+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:25.375 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:25.196+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:25.376 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:25.196+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:25.376 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:25.196+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:25.376 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:25.197+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:25.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:25.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:25.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:25.373+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:25.374+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:26.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:25.724+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:26.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:25.724+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:26.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:25.724+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:26.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:25.725+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:26.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:26.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:26.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:26.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:26.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:26.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:26.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:26.344+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:26.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:26.190+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:26.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:26.190+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:26.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:26.190+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:26.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:26.191+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:26.726+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:26.726+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:26.726+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:26.727+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:27.577 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:27.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:27.577 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:27.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:27.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:27.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:27.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:27.342+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:27.578 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:27.224+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:27.578 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:27.224+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:27.578 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:27.224+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:27.578 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:27.225+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:27.880 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:35:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:35:27.575+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46738 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:35:27.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:27.744+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:27.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:27.744+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:27.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:27.744+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:27.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:27.744+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:28.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:28.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:28.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:28.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:28.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:28.384+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:28.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:28.385+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:28.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:28.271+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:28.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:28.271+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:28.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:28.271+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:28.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:28.272+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:28.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:28.702+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:28.702+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:28.702+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:28.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:28.702+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:29.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:29.247+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:29.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:29.247+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:29.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:29.247+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:29.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:29.247+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:29.371+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:29.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:29.371+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:29.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:29.371+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:29.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:29.371+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:30.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:29.733+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:30.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:29.733+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:30.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:29.733+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:30.135 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:29.734+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:30.359+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:30.359+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:30.359+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:30.359+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:30.242+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:30.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:30.242+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:30.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:30.242+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:30.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:30.242+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:31.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:30.774+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:31.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:30.774+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:31.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:30.774+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:31.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:30.774+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:31.314+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:31.314+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:31.314+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:31.315+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:31.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:31.255+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:31.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:31.255+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:31.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:31.255+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:31.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:31.256+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:32.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:31.801+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:32.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:31.801+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:32.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:31.801+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:32.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:31.802+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:32.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:32.271+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:32.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:32.271+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:32.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:32.271+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:32.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:32.271+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:32.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:32.256+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:32.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:32.256+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:32.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:32.256+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:32.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:32.257+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:32.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:35:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:35:32.581+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46782 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:35:32.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:32.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:32.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:32.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:32.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:32.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:32.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:32.822+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:33.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:33.261+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:33.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:33.261+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:33.707 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:33.261+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:33.707 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:33.262+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:33.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:33.230+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:33.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:33.230+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:33.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:33.230+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:33.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:33.231+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:33.801+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:33.801+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:33.801+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:33.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:33.802+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:34.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:34.188+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:34.188+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:34.188+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:34.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:34.188+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:34.286+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:34.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:34.286+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:34.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:34.286+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:34.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:34.287+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:35.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:34.817+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:35.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:34.817+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:35.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:34.817+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:35.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:34.817+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:35.202+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:35.202+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:35.202+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:35.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:35.203+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:35.244+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:35.244+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:35.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:35.244+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:35.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:35.245+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:36.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:35.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:36.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:35.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:36.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:35.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:36.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:35.823+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:36.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:36.220+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:36.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:36.220+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:36.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:36.220+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:36.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:36.221+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:36.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:36.198+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:36.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:36.198+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:36.198+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:36.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:36.199+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:37.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:36.820+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:37.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:36.820+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:37.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:36.820+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:37.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:36.821+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:37.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:37.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:37.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:37.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:37.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:37.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:37.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:37.238+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:37.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:37.171+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:37.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:37.171+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:37.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:37.171+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:37.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:37.172+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:37.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:35:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:35:37.586+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46828 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:35:37.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:37.800+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:37.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:37.800+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:37.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:37.800+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:37.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:37.801+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:38.241+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:38.241+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:38.241+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:38.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:38.242+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:38.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:38.147+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:38.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:38.147+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:38.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:38.147+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:38.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:38.147+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:39.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:38.842+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:39.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:38.842+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:39.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:38.842+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:39.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:38.843+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:39.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:39.202+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:39.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:39.202+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:39.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:39.202+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:39.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:39.203+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:39.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:39.124+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:39.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:39.124+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:39.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:39.124+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:39.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:39.125+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:40.089+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:40.089+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:40.089+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:40.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:40.090+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:40.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:39.857+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:40.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:39.857+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:40.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:39.857+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:40.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:39.857+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:40.247+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:40.247+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:40.247+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:40.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:40.248+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:41.137 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:40.864+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:41.137 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:40.864+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:41.137 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:40.864+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:41.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:40.865+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:41.217+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:41.217+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:41.217+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:41.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:41.217+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:41.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:41.135+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:41.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:41.135+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:41.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:41.135+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:41.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:41.136+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:42.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:41.831+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:42.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:41.831+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:42.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:41.831+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:42.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:41.832+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:42.256+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:42.256+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:42.256+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:42.257+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:42.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:42.161+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:42.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:42.161+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:42.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:42.161+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:42.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:42.162+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:42.848 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:35:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:35:42.593+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46872 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:35:43.189 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:42.846+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:43.189 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:42.846+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:43.189 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:42.846+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:43.189 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:42.847+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:43.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:43.244+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:43.244+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:43.244+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:43.244+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:43.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:43.186+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:43.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:43.186+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:43.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:43.186+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:43.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:43.187+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:43.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:43.833+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:43.833+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:43.833+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:43.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:43.834+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:44.235+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:44.235+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:44.235+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:44.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:44.236+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:44.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:44.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:44.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:44.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:44.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:44.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:44.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:44.277+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:45.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:44.802+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:44.802+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:44.802+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:45.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:44.802+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:45.267+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:45.267+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:45.267+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:45.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:45.268+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:45.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:45.263+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:45.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:45.263+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:45.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:45.263+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:45.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:45.264+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:46.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:45.756+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:46.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:45.756+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:46.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:45.756+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:46.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:45.756+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:46.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:46.253+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:46.253+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:46.253+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:46.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:46.253+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:46.226+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:46.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:46.226+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:46.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:46.226+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:46.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:46.227+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:47.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:46.731+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:47.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:46.731+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:47.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:46.731+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:47.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:46.732+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:47.243+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:47.243+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:47.243+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:47.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:47.243+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:47.190+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:47.190+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:47.190+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:47.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:47.191+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:47.873 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:35:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:35:47.599+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46919 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:35:47.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:47.772+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:47.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:47.772+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:47.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:47.772+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:47.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:47.773+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:48.191+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:48.191+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:48.191+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:48.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:48.191+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:48.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:48.231+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:48.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:48.231+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:48.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:48.231+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:48.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:48.231+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:49.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:48.766+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:49.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:48.766+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:49.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:48.766+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:49.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:48.767+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:49.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:49.157+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:49.157+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:49.157+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:49.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:49.157+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:49.258+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:49.258+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:49.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:49.258+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:49.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:49.259+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:50.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:49.778+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:50.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:49.778+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:50.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:49.778+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:50.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:49.779+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:50.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:50.174+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:50.174+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:50.174+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:50.175+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:50.300+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:50.300+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:50.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:50.300+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:50.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:50.300+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:51.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:50.818+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:51.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:50.818+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:51.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:50.818+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:51.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:50.819+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:51.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:51.179+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:51.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:51.179+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:51.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:51.179+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:51.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:51.179+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:51.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:51.290+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:51.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:51.290+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:51.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:51.290+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:51.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:51.290+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:52.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:51.841+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:52.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:51.841+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:52.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:51.841+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:52.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:51.842+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:52.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:52.206+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:52.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:52.206+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:52.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:52.206+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:52.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:52.206+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:52.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:52.318+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:52.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:52.318+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:52.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:52.318+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:52.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:52.319+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:52.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:35:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:35:52.605+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 46963 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:35:52.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:52.810+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:52.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:52.810+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:52.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:52.810+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:52.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:52.811+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:53.365+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:53.365+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:53.365+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:53.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:53.366+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:53.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:53.202+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:53.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:53.202+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:53.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:53.202+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:53.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:53.203+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:53.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:53.846+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:53.846+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:53.846+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:53.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:53.847+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:54.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:54.230+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:54.230+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:54.230+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:54.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:54.231+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:54.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:54.395+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:54.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:54.395+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:54.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:54.395+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:54.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:54.395+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:55.026 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-09-20T20:35:55.033 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T20:35:55.033 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-09-20T20:35:55.034 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-09-20T20:35:55.068 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-20T20:35:55.079 INFO:tasks.cephadm:Teardown begin 2024-09-20T20:35:55.080 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephadm.py", line 2374, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-09-20T20:35:55.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T20:35:55.111 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:54.804+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:55.111 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:54.804+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:55.111 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:54.804+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:55.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:54.805+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:55.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:35:55.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:35:55.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:55.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:55.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:55.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:55.362 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:55.255+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:55.362 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:55.255+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:55.362 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:55.255+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:55.362 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:55.256+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:55.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:55.359+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:55.360+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:55.810+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:55.810+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:55.810+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:56.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:55.811+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:56.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:56.286+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:56.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:56.286+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:56.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:56.286+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:56.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:56.287+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:56.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:56.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:56.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:56.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:56.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:56.371+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:56.690 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:56.372+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:56.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:56.828+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:56.828+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:56.828+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:56.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:56.829+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:57.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:57.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:57.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:57.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:57.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:57.328+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:57.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:57.329+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:57.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:57.367+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:57.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:57.367+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:57.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:57.367+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:57.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:57.368+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:57.865 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:35:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:35:57.611+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47009 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:35:57.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:57.793+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:57.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:57.793+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:57.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:57.793+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:57.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:57.794+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:58.191 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:35:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:58.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:58.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:58.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:58.362+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:58.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:58.362+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:35:58.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:58.392+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:58.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:58.392+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:58.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:58.392+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:58.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:58.393+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:58.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:58.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:58.822+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:35:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:58.823+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:59.370+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:59.370+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:59.370+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:35:59.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:35:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:35:59.371+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:35:59.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:59.320+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:59.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:59.320+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:59.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:59.320+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:35:59.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:35:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:35:59.320+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:00.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:59.780+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:00.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:59.780+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:00.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:59.780+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:00.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:35:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:35:59.780+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:00.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:00.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:00.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:00.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:00.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:00.348+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:00.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:00.348+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:00.672 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:00.326+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:00.672 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:00.326+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:00.672 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:00.326+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:00.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:00.327+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:00.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:00.807+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:00.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:00.807+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:00.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:00.807+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:00.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:00.807+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:01.557 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:01.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:01.557 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:01.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:01.557 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:01.378+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:01.557 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:01.378+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:01.557 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:01.294+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:01.558 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:01.294+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:01.558 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:01.294+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:01.558 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:01.295+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:01.845+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:01.845+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:01.845+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:02.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:01.846+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:02.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:02.337+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:02.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:02.337+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:02.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:02.337+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:02.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:02.338+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:02.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:02.337+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:02.622 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:02.337+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:02.622 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:02.337+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:02.622 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:02.338+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:02.886 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:36:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:36:02.618+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47054 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:36:03.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:02.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:02.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:02.884+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:03.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:02.885+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:03.708 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:03.380+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:03.708 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:03.380+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:03.708 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:03.380+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:03.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:03.381+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:03.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:03.338+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:03.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:03.338+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:03.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:03.338+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:03.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:03.339+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:03.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:03.926+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:03.926+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:03.926+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:03.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:03.927+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:04.361+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:04.361+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:04.361+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:04.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:04.362+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:04.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:04.346+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:04.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:04.346+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:04.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:04.346+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:04.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:04.346+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:05.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:04.910+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:04.910+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:04.910+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:05.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:04.911+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:05.367+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:05.367+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:05.367+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:05.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:05.368+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:05.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:05.326+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:05.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:05.326+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:05.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:05.326+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:05.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:05.326+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:05.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:05.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:05.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:05.926+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:06.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:06.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:06.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:06.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:06.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:06.384+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:06.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:06.385+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:06.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:06.300+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:06.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:06.300+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:06.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:06.300+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:06.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:06.300+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:07.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:36:07.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:36:07.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:06.961+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:07.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:06.961+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:07.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:06.961+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:07.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:06.962+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:07.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:07.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:07.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:07.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:07.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:07.315+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:07.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:07.316+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:07.628 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:07.363+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:07.628 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:07.363+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:07.628 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:07.363+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:07.628 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:07.364+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:07.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:36:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:36:07.625+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47101 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:36:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:07.995+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:07.995+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:08.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:07.995+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:08.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:07.996+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:08.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:08.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:08.343+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:08.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:08.343+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:08.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:08.372+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:08.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:08.372+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:08.372+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:08.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:08.373+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:09.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:08.946+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:09.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:08.946+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:09.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:08.946+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:09.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:08.947+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:09.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:09.331+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:09.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:09.331+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:09.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:09.331+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:09.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:09.331+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:09.333+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:09.333+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:09.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:09.333+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:09.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:09.334+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:09.936+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:09.936+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:09.936+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:10.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:09.937+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:10.255 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:36:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:10.316+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:10.316+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:10.316+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:10.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:10.317+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:10.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:10.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:10.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:10.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:10.346+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:10.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:10.347+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:11.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:10.984+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:11.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:10.984+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:11.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:10.984+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:11.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:10.985+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:11.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:11.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:11.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:11.332+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:11.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:11.333+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:11.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:11.339+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:11.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:11.339+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:11.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:11.339+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:11.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:11.339+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:12.299 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:12.030+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:12.299 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:12.030+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:12.300 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:12.030+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:12.300 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:12.030+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:12.633 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:12.350+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:12.634 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:12.350+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:12.634 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:12.350+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:12.634 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:12.350+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:12.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:12.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:12.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:12.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:12.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:12.297+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:12.635 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:12.298+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:12.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:36:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:36:12.631+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47145 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:36:13.301 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:13.009+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:13.301 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:13.009+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:13.301 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:13.009+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:13.301 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:13.010+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:13.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:13.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:13.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:13.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:13.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:13.298+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:13.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:13.299+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:13.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:13.375+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:13.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:13.375+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:13.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:13.375+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:13.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:13.376+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:14.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:14.020+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:14.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:14.020+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:14.311 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:14.020+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:14.311 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:14.021+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:14.423+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:14.423+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:14.423+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:14.424+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:14.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:14.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:14.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:14.308+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:15.332 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:14.984+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:15.332 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:14.984+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:15.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:14.984+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:15.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:14.985+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:15.432+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:15.432+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:15.432+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:15.433+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:15.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:15.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:15.330+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:15.331+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:16.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:15.956+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:16.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:15.956+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:16.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:15.956+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:16.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:15.957+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:16.363+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:16.363+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:16.363+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:16.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:16.364+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:16.430+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:16.430+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:16.430+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:16.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:16.430+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:16.955+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:16.955+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:16.955+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:17.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:16.956+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:17.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:17.381+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:17.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:17.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:17.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:17.382+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:17.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:17.382+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:17.640 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:17.413+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:17.640 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:17.413+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:17.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:17.413+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:17.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:17.414+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:17.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:36:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:36:17.637+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47191 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:36:18.420 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:17.965+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:18.421 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:17.965+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:18.421 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:17.965+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:18.421 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:17.966+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:18.431+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:18.431+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:18.431+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:18.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:18.432+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:18.418+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:18.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:18.418+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:18.418+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:18.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:18.419+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:18.938+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:18.938+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:18.938+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:19.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:18.939+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:19.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:19.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:19.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:19.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:19.472+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:19.385+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:19.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:19.385+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:19.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:19.385+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:19.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:19.386+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:20.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:19.898+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:20.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:19.898+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:20.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:19.898+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:20.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:19.899+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:20.444+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:20.444+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:20.444+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:20.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:20.445+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:20.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:20.354+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:20.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:20.354+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:20.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:20.354+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:20.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:20.355+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:21.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:20.857+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:21.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:20.857+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:21.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:20.857+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:21.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:20.858+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:21.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:21.404+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:21.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:21.404+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:21.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:21.404+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:21.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:21.405+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:21.419+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:21.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:21.419+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:21.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:21.419+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:21.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:21.419+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:21.863+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:21.863+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:21.863+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:22.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:21.864+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:22.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:36:22.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:36:22.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:22.375+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:22.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:22.375+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:22.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:22.375+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:22.647 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:22.375+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:22.647 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:22.425+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:22.647 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:22.425+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:22.647 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:22.425+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:22.647 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:22.426+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:22.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:36:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:36:22.642+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47236 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:36:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:22.865+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:22.865+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:22.865+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:22.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:22.865+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:23.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:23.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:23.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:23.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:23.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:23.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:23.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:23.357+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:23.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:23.399+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:23.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:23.399+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:23.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:23.399+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:23.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:23.400+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:23.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:23.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:23.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:23.890+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:23.890+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:24.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:24.416+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:24.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:24.416+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:24.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:24.416+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:24.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:24.416+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:24.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:24.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:24.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:24.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:24.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:24.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:24.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:24.329+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:25.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:24.882+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:25.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:24.882+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:25.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:24.882+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:25.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:24.882+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:25.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:36:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:25.365+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:25.365+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:25.365+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:25.366+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:25.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:25.435+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:25.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:25.435+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:25.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:25.435+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:25.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:25.436+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:26.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:25.881+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:26.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:25.881+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:26.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:25.881+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:26.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:25.882+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:26.364+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:26.364+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:26.364+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:26.364+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:26.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:26.477+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:26.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:26.477+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:26.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:26.477+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:26.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:26.478+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:26.885+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:26.885+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:26.885+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:27.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:26.886+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:27.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:27.349+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:27.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:27.349+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:27.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:27.349+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:27.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:27.350+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:27.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:27.434+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:27.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:27.434+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:27.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:27.434+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:27.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:27.434+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:27.927 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:36:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:36:27.649+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47282 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:36:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:27.924+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:27.924+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:27.924+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:28.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:27.925+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:28.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:36:28.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:36:28.398 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:36:28.399 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-20T20:36:28.399 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-20T20:36:28.399 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-20T20:36:28.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:36:28.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:36:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:28.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:28.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:28.478+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:28.479+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:28.322+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:28.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:28.322+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:28.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:28.322+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:28.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:28.323+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:29.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:28.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:29.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:28.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:29.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:28.925+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:29.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:28.926+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:29.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:29.305+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:29.305+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:29.305+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:29.306+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:29.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:29.472+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:29.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:29.472+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:29.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:29.472+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:29.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:29.473+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:30.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:29.973+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:30.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:29.973+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:30.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:29.973+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:30.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:29.973+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:30.270+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:30.270+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:30.270+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:30.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:30.271+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:30.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:30.481+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:30.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:30.481+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:30.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:30.481+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:30.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:30.482+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:31.003+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:31.003+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:31.003+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:31.003+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:31.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:31.243+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:31.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:31.243+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:31.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:31.243+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:31.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:31.244+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:31.474 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:36:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:31.519+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:31.519+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:31.519+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:31.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:31.519+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:32.026+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:32.026+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:32.026+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:32.026+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:32.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:32.199+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:32.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:32.199+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:32.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:32.199+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:32.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:32.199+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:32.860 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:36:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:36:32.656+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47326 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:36:32.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:32.566+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:32.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:32.566+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:32.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:32.566+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:32.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:32.567+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:33.060+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:33.060+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:33.060+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:33.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:33.061+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:33.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:33.219+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:33.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:33.219+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:33.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:33.219+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:33.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:33.219+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:33.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:33.581+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:33.581+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:33.581+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:33.582+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:34.025+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:34.025+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:34.025+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:34.026+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:34.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:34.190+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:34.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:34.190+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:34.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:34.190+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:34.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:34.190+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:34.559+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:34.559+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:34.559+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:34.560+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:34.983+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:34.983+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:34.983+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:34.984+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:35.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:35.219+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:35.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:35.219+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:35.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:35.219+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:35.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:35.220+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:35.946 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:35.579+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:35.947 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:35.579+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:35.947 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:35.579+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:35.947 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:35.580+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:36.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:35.944+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:36.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:35.944+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:36.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:35.944+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:36.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:35.945+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:36.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:36.256+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:36.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:36.256+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:36.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:36.256+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:36.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:36.256+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:36.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:36.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:36.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:36.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:36.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:36.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:36.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:36.577+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:37.308 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:36.977+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:37.308 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:36.977+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:37.308 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:36.977+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:37.309 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:36.978+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:37.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:37.306+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:37.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:37.306+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:37.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:37.306+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:37.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:37.306+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:37.876 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:36:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:36:37.662+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47372 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:36:37.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:37.601+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:37.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:37.601+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:37.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:37.601+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:37.877 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:37.602+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:38.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:38.022+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:38.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:38.022+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:38.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:38.022+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:38.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:38.023+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:38.649 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:38.321+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:38.649 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:38.321+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:38.649 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:38.321+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:38.649 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:38.322+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:38.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:38.646+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:38.646+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:38.646+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:38.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:38.648+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:39.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:39.009+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:39.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:39.009+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:39.311 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:39.009+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:39.311 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:39.009+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:39.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:39.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:39.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:39.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:39.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:39.308+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:39.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:39.309+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:39.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:39.662+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:39.662+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:39.662+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:39.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:39.663+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:40.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:40.261+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:40.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:40.261+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:40.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:40.261+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:40.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:40.261+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:40.298 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:40.037+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:40.298 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:40.037+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:40.298 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:40.037+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:40.299 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:40.038+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:40.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:36:40.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:36:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:40.677+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:40.677+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:40.677+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:40.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:40.678+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:41.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:41.233+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:41.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:41.233+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:41.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:41.233+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:41.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:41.234+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:41.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:41.055+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:41.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:41.055+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:41.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:41.055+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:41.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:41.055+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:41.633+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:41.633+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:41.633+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:41.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:41.633+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:42.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:42.267+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:42.267+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:42.267+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:42.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:42.268+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:42.095+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:42.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:42.095+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:42.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:42.095+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:42.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:42.096+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:36:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:36:42.668+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47417 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:36:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:42.675+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:42.675+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:42.675+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:42.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:42.676+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:43.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:43.252+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:43.252+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:43.252+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:43.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:43.253+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:43.116+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:43.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:43.116+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:43.116+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:43.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:43.116+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:43.535 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:36:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:43.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:43.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:43.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:43.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:43.666+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:44.225+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:44.225+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:44.225+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:44.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:44.225+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:44.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:44.163+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:44.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:44.163+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:44.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:44.163+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:44.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:44.164+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:44.651+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:44.651+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:44.651+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:44.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:44.652+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:45.193+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:45.193+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:45.193+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:45.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:45.193+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:45.210+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:45.210+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:45.210+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:45.211+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:45.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:45.626+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:45.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:45.626+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:45.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:45.626+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:45.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:45.627+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:46.608 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:46.214+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:46.608 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:46.214+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:46.608 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:46.214+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:46.608 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:46.215+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:46.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:46.240+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:46.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:46.240+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:46.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:46.240+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:46.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:46.241+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:46.605+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:46.605+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:46.605+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:46.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:46.606+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:47.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:47.187+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:47.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:47.187+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:47.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:47.187+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:47.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:47.188+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:47.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:47.236+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:47.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:47.236+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:47.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:47.236+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:47.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:47.236+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:47.868 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:36:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:36:47.674+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47463 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:36:47.868 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:47.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:47.868 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:47.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:47.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:47.576+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:47.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:47.577+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:48.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:48.224+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:48.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:48.224+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:48.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:48.224+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:48.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:48.224+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:48.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:48.281+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:48.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:48.281+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:48.629 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:48.281+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:48.629 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:48.282+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:48.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:48.625+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:48.625+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:48.625+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:48.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:48.626+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:49.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:49.262+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:49.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:49.262+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:49.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:49.262+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:49.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:49.262+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:49.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:49.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:49.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:49.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:49.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:49.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:49.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:49.322+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:49.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:49.638+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:49.638+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:49.638+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:49.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:49.639+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:50.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:50.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:50.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:50.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:50.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:50.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:50.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:50.284+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:50.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:50.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:50.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:50.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:50.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:50.237+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:50.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:50.237+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:50.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:50.633+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:50.633+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:50.633+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:50.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:50.634+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:51.544 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:51.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:51.544 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:51.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:51.545 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:51.277+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:51.545 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:51.277+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:51.545 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:51.289+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:51.545 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:51.289+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:51.545 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:51.289+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:51.545 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:51.290+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:51.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:51.587+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:51.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:51.588+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:51.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:51.588+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:52.571 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:52.314+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:52.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:52.314+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:52.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:52.314+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:52.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:52.315+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:52.572 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:52.309+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:52.573 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:52.309+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:52.573 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:52.309+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:52.573 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:52.309+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:52.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:36:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:36:52.679+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47507 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:36:52.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:52.569+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:52.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:52.569+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:52.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:52.569+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:52.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:52.570+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:53.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:53.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:53.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:53.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:53.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:53.329+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:53.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:53.329+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:53.600 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:53.310+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:53.600 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:53.310+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:53.600 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:53.310+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:53.600 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:53.311+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:53.597+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:53.597+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:53.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:53.597+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:53.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:53.597+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:54.587 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:54.313+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:54.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:54.313+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:54.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:54.313+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:54.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:54.314+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:54.588 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:54.313+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:54.588 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:54.313+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:54.589 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:54.313+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:54.589 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:54.313+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:54.585+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:54.585+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:54.585+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:54.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:54.586+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:55.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:36:55.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:36:55.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:55.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:55.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:55.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:55.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:55.344+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:55.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:55.345+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:55.597 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:55.342+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:55.597 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:55.342+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:55.597 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:55.342+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:55.597 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:55.342+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:55.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:55.593+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:55.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:55.593+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:55.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:55.593+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:55.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:55.594+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:56.632 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:56.375+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:56.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:56.375+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:56.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:56.375+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:56.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:56.376+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:56.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:56.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:56.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:56.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:56.634 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:56.321+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:56.634 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:56.322+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:56.634 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:56.553+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:56.634 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:56.553+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:56.634 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:56.553+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:56.634 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:56.554+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:57.592 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:57.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:57.592 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:57.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:57.593 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:57.345+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:57.593 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:57.346+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:57.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:57.294+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:57.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:57.294+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:57.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:57.294+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:57.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:57.295+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:57.870 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:36:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:36:57.685+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47553 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:36:57.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:57.590+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:57.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:57.590+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:57.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:57.590+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:57.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:57.590+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:58.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:58.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:58.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:58.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:58.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:58.319+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:58.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:58.320+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:58.596 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:58.248+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:58.596 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:58.248+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:58.596 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:58.248+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:58.596 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:58.249+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:58.606 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:36:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:58.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:58.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:58.592+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:58.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:58.593+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:59.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:59.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:59.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:59.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:59.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:59.341+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:36:59.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:36:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:36:59.342+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:36:59.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:59.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:59.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:59.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:59.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:59.199+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:36:59.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:36:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:36:59.200+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:36:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:59.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:59.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:59.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:36:59.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:36:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:36:59.576+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:00.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:00.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:00.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:00.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:00.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:00.356+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:00.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:00.356+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:00.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:00.204+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:00.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:00.204+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:00.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:00.204+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:00.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:00.205+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:00.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:00.551+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:00.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:00.551+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:00.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:00.551+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:00.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:00.552+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:01.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:01.395+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:01.395+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:01.395+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:01.396+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:01.170+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:01.170+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:01.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:01.170+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:01.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:01.171+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:01.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:37:01.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:37:01.679 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:37:01.679 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-20T20:37:01.679 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-20T20:37:01.681 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi/7912799/remote/smithi114/crash 2024-09-20T20:37:01.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/crash -- . 2024-09-20T20:37:01.709 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi/7912799/remote/smithi160/crash 2024-09-20T20:37:01.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:37:01.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:37:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:01.570+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:01.570+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:01.570+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:01.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:01.571+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:02.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:02.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:02.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:02.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:02.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:02.154+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:02.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:02.155+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:02.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:02.421+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:02.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:02.421+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:02.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:02.421+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:02.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:02.422+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:02.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:02.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:02.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:02.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:02.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:02.575+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:02.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:02.576+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:02.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:37:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:37:02.692+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47598 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:37:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:03.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:03.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:03.401+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:03.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:03.402+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:03.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:03.190+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:03.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:03.190+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:03.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:03.190+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:03.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:03.191+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:03.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:03.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:03.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:03.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:03.618+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:04.620 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:04.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:04.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:04.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:04.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:04.428+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:04.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:04.428+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:04.622 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:04.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:04.622 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:04.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:04.622 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:04.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:04.622 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:04.238+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:04.783 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:37:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:04.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:04.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:04.618+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:04.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:04.619+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:05.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:05.406+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:05.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:05.406+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:05.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:05.406+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:05.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:05.407+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:05.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:05.284+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:05.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:05.284+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:05.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:05.284+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:05.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:05.284+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:05.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:05.609+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:05.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:05.609+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:05.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:05.609+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:05.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:05.610+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:06.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:06.367+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:06.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:06.367+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:06.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:06.367+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:06.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:06.367+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:06.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:06.305+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:06.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:06.305+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:06.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:06.305+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:06.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:06.306+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:06.640+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:06.640+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:06.640+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:06.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:06.641+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:07.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:07.324+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:07.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:07.324+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:07.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:07.324+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:07.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:07.325+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:07.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:07.347+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:07.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:07.347+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:07.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:07.347+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:07.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:07.348+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:07.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:37:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:37:07.697+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47644 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:37:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:07.656+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:07.656+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:07.656+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:07.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:07.656+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:08.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:08.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:08.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:08.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:08.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:08.342+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:08.653 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:08.343+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:08.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:08.336+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:08.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:08.336+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:08.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:08.336+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:08.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:08.337+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:08.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:08.649+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:08.649+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:08.649+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:08.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:08.649+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:09.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:09.339+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:09.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:09.339+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:09.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:09.339+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:09.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:09.340+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:09.670 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:09.297+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:09.670 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:09.297+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:09.670 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:09.297+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:09.670 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:09.298+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:09.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:09.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:09.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:09.666+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:09.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:09.666+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:10.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:10.372+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:10.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:10.372+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:10.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:10.372+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:10.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:10.372+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:10.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:10.249+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:10.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:10.249+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:10.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:10.249+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:10.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:10.250+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:10.672+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:10.672+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:10.672+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:10.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:10.673+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:11.664 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:11.407+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:11.664 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:11.407+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:11.664 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:11.407+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:11.664 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:11.408+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:11.664 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:11.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:11.665 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:11.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:11.665 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:11.283+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:11.665 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:11.283+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:11.920 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:11.661+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:11.920 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:11.661+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:11.920 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:11.661+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:11.920 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:11.662+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:12.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:12.457+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:12.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:12.457+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:12.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:12.457+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:12.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:12.457+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:12.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:12.333+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:12.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:12.333+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:12.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:12.333+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:12.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:12.334+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:12.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:37:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:37:12.704+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47688 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:37:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:12.643+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:12.643+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:12.643+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:12.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:12.644+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:13.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:13.456+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:13.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:13.456+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:13.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:13.456+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:13.623 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:13.457+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:13.624 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:13.311+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:13.624 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:13.311+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:13.624 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:13.311+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:13.624 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:13.312+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:13.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:37:13.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:37:13.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:13.620+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:13.620+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:13.620+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:13.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:13.621+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:14.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:14.448+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:14.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:14.448+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:14.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:14.448+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:14.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:14.449+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:14.655 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:14.310+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:14.655 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:14.310+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:14.656 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:14.310+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:14.656 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:14.311+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:14.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:14.652+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:14.652+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:14.652+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:14.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:14.653+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:15.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:15.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:15.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:15.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:15.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:15.471+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:15.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:15.471+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:15.692 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:15.282+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:15.692 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:15.282+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:15.692 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:15.282+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:15.692 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:15.283+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:15.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:15.689+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:15.689+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:15.689+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:15.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:15.689+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:16.561 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:16.449+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:16.561 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:16.449+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:16.562 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:16.449+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:16.562 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:16.450+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:16.562 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:16.255+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:16.562 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:16.255+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:16.562 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:16.255+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:16.562 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:16.255+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:16.846 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:37:16.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:16.735+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:16.735+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:16.735+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:16.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:16.736+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:17.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:17.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:17.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:17.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:17.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:17.440+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:17.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:17.441+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:17.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:17.253+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:17.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:17.253+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:17.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:17.253+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:17.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:17.254+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:17.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:37:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:37:17.711+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47734 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:37:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:17.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:17.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:17.748+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:17.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:17.748+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:18.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:18.405+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:18.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:18.405+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:18.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:18.405+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:18.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:18.406+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:18.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:18.226+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:18.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:18.226+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:18.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:18.226+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:18.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:18.227+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:18.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:18.699+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:18.699+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:18.699+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:18.700+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:19.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:19.410+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:19.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:19.410+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:19.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:19.410+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:19.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:19.410+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:19.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:19.210+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:19.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:19.210+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:19.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:19.210+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:19.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:19.210+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:20.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:19.745+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:20.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:19.745+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:20.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:19.745+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:20.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:19.746+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:20.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:20.427+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:20.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:20.427+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:20.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:20.427+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:20.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:20.428+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:20.204+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:20.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:20.204+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:20.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:20.204+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:20.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:20.205+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:21.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:20.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:21.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:20.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:21.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:20.783+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:21.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:20.784+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:21.421+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:21.422+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:21.422+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:21.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:21.422+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:21.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:21.197+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:21.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:21.197+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:21.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:21.197+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:21.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:21.198+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:22.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:21.753+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:22.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:21.753+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:22.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:21.753+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:22.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:21.754+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:22.445 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:22.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:22.445 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:22.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:22.445 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:22.237+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:22.445 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:22.238+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:22.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:22.442+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:22.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:22.442+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:22.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:22.442+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:22.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:22.443+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:22.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:37:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:37:22.719+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47779 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:37:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:22.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:22.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:22.727+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:22.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:22.728+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:23.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:23.475+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:23.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:23.475+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:23.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:23.475+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:23.714 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:23.476+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:23.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:23.244+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:23.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:23.244+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:23.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:23.244+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:23.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:23.245+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:23.736+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:23.736+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:23.736+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:23.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:23.737+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:24.267+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:24.267+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:24.267+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:24.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:24.268+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:24.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:24.485+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:24.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:24.485+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:24.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:24.485+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:24.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:24.486+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:25.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:24.747+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:24.747+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:24.747+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:25.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:24.748+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:25.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:25.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:25.473+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:25.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:25.473+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:25.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:25.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:25.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:25.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:25.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:25.272+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:25.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:25.272+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:25.724+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:25.724+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:25.724+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:26.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:25.725+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:26.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:26.233+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:26.233+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:26.233+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:26.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:26.234+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:26.446+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:26.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:26.446+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:26.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:26.446+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:26.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:26.447+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:27.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:26.753+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:27.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:26.753+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:27.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:26.753+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:27.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:26.753+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:27.455 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:27.452+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:27.455 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:27.452+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:27.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:27.452+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:27.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:27.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:27.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:27.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:27.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:27.203+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:27.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:27.204+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:27.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:27.453+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:28.216 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:37:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:37:27.726+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47825 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:37:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:27.764+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:27.764+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:27.764+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:28.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:27.765+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:28.239+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:28.239+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:28.239+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:28.240+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:28.422+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:28.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:28.422+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:28.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:28.422+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:28.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:28.423+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:28.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:37:28.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:37:29.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:28.761+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:28.761+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:28.761+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:29.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:28.762+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:29.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:29.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:29.387+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:29.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:29.388+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:29.229+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:29.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:29.229+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:29.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:29.229+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:29.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:29.230+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:30.190 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:29.763+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:30.190 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:29.763+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:30.190 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:29.763+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:30.190 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:29.764+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:30.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:30.395+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:30.395+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:30.395+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:30.396+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:30.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:30.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:30.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:30.187+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:30.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:30.188+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:31.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:30.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:31.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:30.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:31.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:30.767+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:31.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:30.768+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:31.443 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:31.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:31.443 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:31.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:31.443 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:31.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:31.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:31.177+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:31.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:31.441+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:31.441+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:31.441+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:31.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:31.442+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:31.922 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:37:32.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:31.793+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:31.793+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:31.793+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:31.793+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:32.217+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:32.217+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:32.217+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:32.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:32.218+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:32.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:32.436+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:32.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:32.436+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:32.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:32.436+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:32.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:32.437+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:33.177 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:37:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:37:32.732+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47869 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:37:33.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:32.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:33.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:32.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:33.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:32.821+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:33.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:32.822+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:33.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:33.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:33.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:33.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:33.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:33.175+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:33.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:33.176+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:33.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:33.443+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:33.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:33.443+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:33.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:33.443+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:33.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:33.444+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:33.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:33.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:33.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:33.833+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:33.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:33.834+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:34.127+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:34.127+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:34.127+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:34.128+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:34.820 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:34.477+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:34.820 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:34.477+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:34.820 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:34.477+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:34.820 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:34.478+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:34.923 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:37:34.924 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:37:34.994 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:37:34.995 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-20T20:37:34.995 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-20T20:37:34.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-09-20T20:37:35.042 INFO:tasks.cephadm:Compressing logs... 2024-09-20T20:37:35.043 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-20T20:37:35.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:37:35.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:37:35.067 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-20T20:37:35.067 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-osd.0.log 2024-09-20T20:37:35.067 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-mds.cephfs.smithi114.scsmgq.log 2024-09-20T20:37:35.068 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-client.ceph-exporter.smithi114.log 2024-09-20T20:37:35.068 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-osd.1.log 2024-09-20T20:37:35.069 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-mds.cephfs.smithi114.scsmgq.log: /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-client.ceph-exporter.smithi114.log: gzip -5 --verbose -- /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-mds.cephfs.smithi114.tlrcnf.log 2024-09-20T20:37:35.069 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-20T20:37:35.069 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-mds.cephfs.smithi114.tlrcnf.log: gzip -5 --verbose -- /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-20T20:37:35.073 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-volume.log: 94.7% -- replaced with /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-client.ceph-exporter.smithi114.log.gz 2024-09-20T20:37:35.074 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-mgr.smithi114.rvhgji.log 2024-09-20T20:37:35.076 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph.audit.log: 90.7% -- replaced with /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-20T20:37:35.077 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-20T20:37:35.081 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-mgr.smithi114.rvhgji.log: 91.1% -- replaced with /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-20T20:37:35.081 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-mon.smithi114.log 2024-09-20T20:37:35.082 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph.cephadm.log: 84.5% -- replaced with /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-20T20:37:35.083 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-osd.2.log 2024-09-20T20:37:35.097 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-mon.smithi114.log: gzip -5 --verbose -- /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph.log 2024-09-20T20:37:35.110 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-20T20:37:35.111 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph.log: 87.2% -- replaced with /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-20T20:37:35.120 INFO:teuthology.orchestra.run.smithi114.stderr: 95.9% -- replaced with /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-mds.cephfs.smithi114.tlrcnf.log.gz 2024-09-20T20:37:35.171 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/cephadm.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-20T20:37:35.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:34.817+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:35.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:34.817+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:35.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:34.817+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:35.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:34.818+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:35.437 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:35.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:35.437 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:35.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:35.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:35.435+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:26:58.685563+0000 (oldest deadline 2024-09-20T20:27:18.685563+0000) 2024-09-20T20:37:35.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:35.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:35.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:35.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:35.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:35.174+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:35.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:35.174+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:35.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:35.435+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:36.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:35.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:36.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:35.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:36.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:35.780+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:36.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:35.781+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:36.306 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-mgr.smithi114.rvhgji.log: file size changed while zipping 2024-09-20T20:37:36.306 INFO:teuthology.orchestra.run.smithi114.stderr: 90.0% -- replaced with /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-mgr.smithi114.rvhgji.log.gz 2024-09-20T20:37:36.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:36.449+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:36.454 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:36.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:36.454 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:36.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:36.454 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:36.139+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:36.454 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:36.139+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:37.124 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:37.120+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:37.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:36.800+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:37.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:36.800+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:37.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:36.800+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:37.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:36.801+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:37.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:37.120+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:37.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:37.120+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:37.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:37.120+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:37.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:37.427+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:38.127 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:37:38.152 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:37:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:37:37.745+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47915 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:37:38.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:37.777+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:38.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:37.777+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:38.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:37.777+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:00.720382+0000 (oldest deadline 2024-09-20T20:27:20.720382+0000) 2024-09-20T20:37:38.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:37.777+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:38.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:38.149+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:38.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:38.149+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:38.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:38.149+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:38.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:38.151+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:38.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:38.447+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:39.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:38.795+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:39.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:39.161+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:39.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:39.161+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:39.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:39.161+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:39.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:39.162+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:39.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:39.442+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:39.911 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-mon.smithi114.log: file size changed while zipping 2024-09-20T20:37:39.911 INFO:teuthology.orchestra.run.smithi114.stderr: 96.9% -- replaced with /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-mon.smithi114.log.gz 2024-09-20T20:37:40.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:39.747+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:40.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:40.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:40.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:40.188+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:40.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:40.190+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:40.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:40.485+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:41.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:40.786+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:41.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:41.448+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:41.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:41.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:41.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:41.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:41.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:41.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:41.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:41.215+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:42.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:41.812+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:42.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:42.464+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:42.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:42.241+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:42.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:42.241+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:42.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:42.241+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:42.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:42.242+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:42.837 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:37:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:37:42.757+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 47960 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:37:43.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:42.834+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:43.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:43.242+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:43.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:43.242+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:43.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:43.242+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:43.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:43.243+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:43.876 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:43.508+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:43.877 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:43.873+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:44.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:44.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:44.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:44.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:44.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:44.214+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:44.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:44.215+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:44.913 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:44.506+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:45.189 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:44 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:44.910+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:45.186+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:45.186+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:45.186+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:45.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:45.188+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:45.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:45.510+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:46.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:45 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:45.928+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:46.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:46.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:46.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:46.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:46.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:46.193+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:27:13.196536+0000 (oldest deadline 2024-09-20T20:27:33.196536+0000) 2024-09-20T20:37:46.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:46.194+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:46.905 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:46.547+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:46.906 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:46 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:46.902+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:47.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:37:47.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:37:47.580 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:47.232+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:47.863 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:47.577+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:47.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:37:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:37:47.763+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 48006 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:37:48.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:47 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:47.861+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:48.222 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-osd.1.log: file size changed while zipping 2024-09-20T20:37:48.222 INFO:teuthology.orchestra.run.smithi114.stderr: 96.4% -- replaced with /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-osd.1.log.gz 2024-09-20T20:37:48.576 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:48.221+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:48.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:48.574+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:49.132 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-osd.2.log: file size changed while zipping 2024-09-20T20:37:49.132 INFO:teuthology.orchestra.run.smithi114.stderr: 96.4% -- replaced with /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-osd.2.log.gz 2024-09-20T20:37:49.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:48 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:48.900+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:49.563 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:49.263+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:49.734 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-osd.0.log: file size changed while zipping 2024-09-20T20:37:49.735 INFO:teuthology.orchestra.run.smithi114.stderr: 96.4% -- replaced with /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-osd.0.log.gz 2024-09-20T20:37:49.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:49.561+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:50.191 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:37:50.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:49 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:49.947+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:50.591 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:50.238+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:50.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:50.588+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:51.233 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:50 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:50.978+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:51.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:51.230+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:51.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:51.601+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:52.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:51 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:51.941+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:52.654 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:52.269+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:52.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:52.650+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:52.927 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:37:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:37:52.770+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 48050 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:37:53.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:52 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:52.923+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:53.642 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:53.230+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:53.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:53.638+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:54.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:53 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:53.954+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:54.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:54.256+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:54.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:54.644+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:55.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:54 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:54.995+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:55.681 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:55.264+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:55.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:55.678+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:56.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:55 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:55.986+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:56.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:56.310+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:57.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:56.728+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:57.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:56 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:56.975+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:57.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:57.338+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:58.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:37:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:37:57.777+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 48096 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:37:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:57 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:57.737+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:58.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:58.019+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:58.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:58.335+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:59.062 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:58 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:58.781+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:37:59.318 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:37:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:37:59.060+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:37:59.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:37:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:37:59.315+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:00.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:59.743+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:00.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:59.743+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:00.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:59.743+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:00.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:37:59 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:37:59.744+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:00.280 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:00.024+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:00.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:00.278+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:01.056 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:00.775+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:01.057 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:00.775+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:01.057 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:00.775+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:01.057 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:00 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:00.776+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:01.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:01.054+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:01.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:01.308+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:02.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:01.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:02.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:01.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:02.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:01.727+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:02.097 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:01 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:01.728+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:02.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:38:02.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:38:02.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:02.274+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:02.094+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:02.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:38:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:38:02.783+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 48141 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:38:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:02.684+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:02.684+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:02.684+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:02.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:02 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:02.684+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:03.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:03.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:03.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:03.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:03.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:03.081+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:03.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:03.082+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:03.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:03.269+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:03.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:03.646+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:03.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:03.646+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:03.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:03.646+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:03.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:03 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:03.648+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:04.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:04.249+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:04.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:04.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:04.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:04.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:04.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:04.037+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:04.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:04.038+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:04.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:04.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:04.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:04.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:04.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:04.672+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:04.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:04 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:04.673+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:05.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:38:05.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:05.279+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:05.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:05.088+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:05.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:05.088+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:05.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:05.088+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:05.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:05.089+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:05.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:05.642+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:05.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:05.642+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:05.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:05.642+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:05.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:05 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:05.642+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:06.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:06.245+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:06.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:06.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:06.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:06.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:06.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:06.113+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:06.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:06.113+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:06.619+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:06.619+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:06.619+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:06.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:06 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:06.620+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:07.137 INFO:teuthology.orchestra.run.smithi114.stderr: 92.5% -- replaced with /var/log/ceph/6a49c7bc-7787-11ef-baf5-efdc52797490/ceph-mds.cephfs.smithi114.scsmgq.log.gz 2024-09-20T20:38:07.139 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-20T20:38:07.139 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m32.082s 2024-09-20T20:38:07.139 INFO:teuthology.orchestra.run.smithi114.stderr:user 1m17.925s 2024-09-20T20:38:07.139 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m1.890s 2024-09-20T20:38:07.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:07.246+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:07.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:07.246+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:07.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:07.247+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:07.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:07.247+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:07.130+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:07.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:07.130+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:07.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:07.130+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:07.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:07.130+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:07.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:38:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:38:07.789+0000 7f98ac414640 -1 mon.smithi114@0(probing) e2 get_health_metrics reporting 48187 slow ops, oldest is log(1 entries from seq 834 at 2024-09-20T19:44:46.801834+0000) 2024-09-20T20:38:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:07.644+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:07.644+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:07.644+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:07.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:07 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:07.644+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:08.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:08.145+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:08.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:08.145+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:08.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:08.145+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:08.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:08.146+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:08.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:38:08.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:38:08.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:38:08.335 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-20T20:38:08.335 INFO:tasks.cephadm:Removing cluster... 2024-09-20T20:38:08.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 --force 2024-09-20T20:38:08.560 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: 6a49c7bc-7787-11ef-baf5-efdc52797490 2024-09-20T20:38:08.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:08.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:08.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:08.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:08.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:08.228+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:08.647 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:08.229+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:08.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:38:08 smithi114 systemd[1]: Stopping Ceph mon.smithi114 for 6a49c7bc-7787-11ef-baf5-efdc52797490... 2024-09-20T20:38:08.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:08.666+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:08.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:08.666+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:08.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:08.666+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:08.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:08.667+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:09.262 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:38:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:38:08.966+0000 7f98af41a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi114 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:38:09.262 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:38:08 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114[28256]: 2024-09-20T20:38:08.966+0000 7f98af41a640 -1 mon.smithi114@0(probing) e2 *** Got Signal Terminated *** 2024-09-20T20:38:09.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:09.112+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:09.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:09.112+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:09.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:09.112+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:09.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:09.113+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:09.557 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:09.259+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:09.557 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:09.259+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:09.557 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:09.259+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:09.557 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:09.260+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:09.844 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:38:09 smithi114 podman[87578]: 2024-09-20 20:38:09.555689454 +0000 UTC m=+0.739205600 container died ed7f013f251354937f820546027918cc28565eae298d1e9671d882c09e8fcd7f (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-ed8ee0f, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-20T20:38:09.845 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:09.686+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:09.845 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:09.686+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:09.845 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:09.686+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:09.845 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:09 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:09.687+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:10.104 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:38:10 smithi114 podman[87578]: 2024-09-20 20:38:10.05101821 +0000 UTC m=+1.234534361 container cleanup ed7f013f251354937f820546027918cc28565eae298d1e9671d882c09e8fcd7f (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-ed8ee0f, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-20T20:38:10.104 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:38:10 smithi114 bash[87578]: ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114 2024-09-20T20:38:10.452 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:38:10 smithi114 podman[87589]: 2024-09-20 20:38:10.336060704 +0000 UTC m=+0.772198063 container remove ed7f013f251354937f820546027918cc28565eae298d1e9671d882c09e8fcd7f (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-mon-smithi114, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-ed8ee0f, GIT_CLEAN=True) 2024-09-20T20:38:10.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:10.305+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:10.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:10.305+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:10.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:10.305+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:10.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:10.306+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:10.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:10.101+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:10.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:10.101+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:10.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:10.101+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:10.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:10.102+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:10.753 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:38:10 smithi114 systemd[1]: ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@mon.smithi114.service: Deactivated successfully. 2024-09-20T20:38:10.754 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:38:10 smithi114 systemd[1]: Stopped Ceph mon.smithi114 for 6a49c7bc-7787-11ef-baf5-efdc52797490. 2024-09-20T20:38:10.754 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Sep 20 20:38:10 smithi114 systemd[1]: ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@mon.smithi114.service: Consumed 14min 37.587s CPU time. 2024-09-20T20:38:10.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:10.689+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:10.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:10.689+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:10.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:10.689+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:10.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:10 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:10.690+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:11.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:11.062+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:11.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:11.062+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:11.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:11.062+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:11.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:11.063+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:11.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:11.317+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:11.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:11.317+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:11.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:11.317+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:11.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:11.318+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:11.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:11.699+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:11.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:11.699+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:11.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:11.699+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:11.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:11 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:11.699+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:12.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:12.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:12.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:12.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:12.037+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:12.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:12.330+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:12.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:12.330+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:12.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:12.330+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:12.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:12.331+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:12.864 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:12.706+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:12.865 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:12.706+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:12.865 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:12.706+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:12.865 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:12 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:12.707+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:13.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:13.007+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:13.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:13.007+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:13.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:13.007+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:13.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:13.008+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:13.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:13.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:13.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:13.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:13.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:13.365+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:13.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:13.366+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:13.869 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:13.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:13.869 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:13.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:13.869 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:13.740+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:13.869 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:13 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:13.740+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:14.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:14.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:14.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:14.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:14.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:14.036+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:14.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:14.037+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:14.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:14.396+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:14.396+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:14.396+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:14.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:14.396+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:14.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:14.759+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:14.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:14.759+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:14.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:14.759+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:14.799 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:14 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:14.760+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:15.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:15.072+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:15.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:15.072+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:15.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:15.072+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:15.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:15.073+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:15.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:15.363+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:15.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:15.363+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:15.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:15.363+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:15.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:15.364+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:15.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:15.725+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:15.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:15.725+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:15.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:15.725+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:15.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:15 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:15.726+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:16.460 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:16.329+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:16.461 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:16.329+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:16.461 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:16.329+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:16.461 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:16.330+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:16.461 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:16.085+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:16.461 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:16.085+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:16.461 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:16.085+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:16.462 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:16.085+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:17.064 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:16.739+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:17.064 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:16.739+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:17.064 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:16.739+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:17.064 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:16 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:16.740+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:17.319 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:17.061+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:17.319 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:17.061+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:17.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:17.061+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:17.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:17.062+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:17.317+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:17.317+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:17.317+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:17.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:17.318+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:18.048 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:17.789+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:18.048 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:17.789+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:18.048 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:17.789+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:18.048 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:17 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:17.789+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:18.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:18.045+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:18.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:18.045+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:18.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:18.045+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:18.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:18.046+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:18.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:18.303+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:18.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:18.303+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:18.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:18.303+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:18.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:18.304+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:18.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:18.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:18.748+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:18 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:18.749+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:19.277 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:19.261+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:19.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:19.261+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:19.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:19.261+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:19.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:19.261+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:19.279 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:19.093+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:19.279 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:19.093+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:19.279 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:19.093+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:19.280 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:19.094+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:19.948 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:19.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:19.948 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:19.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:19.948 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:19.719+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:19.948 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:19 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:19.720+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:20.104+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:20.104+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:20.104+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:20.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:20.105+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:20.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:20.244+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:20.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:20.244+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:20.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:20.244+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:20.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:20.245+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:20.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:20.688+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:20.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:20.688+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:20.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:20.688+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:20.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:20 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:20.689+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:21.140+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:21.140+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:21.140+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:21.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:21.141+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:21.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:21.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:21.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:21.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:21.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:21.264+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:21.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:21.265+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:21.653+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:21.653+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:21.653+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:21 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:21.653+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:21.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:21 smithi114 systemd[1]: Stopping Ceph osd.0 for 6a49c7bc-7787-11ef-baf5-efdc52797490... 2024-09-20T20:38:22.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:22.005+0000 7f1377cbb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:38:22.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:22.005+0000 7f1377cbb640 -1 osd.0 46 *** Got signal Terminated *** 2024-09-20T20:38:22.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:22.005+0000 7f1377cbb640 -1 osd.0 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:38:22.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:22.253+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:22.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:22.253+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:22.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:22.253+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:22.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:22.253+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:22.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:22.165+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:22.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:22.165+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:22.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:22.165+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:22.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:22.165+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:22.657+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:22.657+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:22.657+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:22.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:22 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:22.658+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:23.281+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:23.281+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:23.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:23.281+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:23.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:23.282+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:23.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:23.164+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:23.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:23.164+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:23.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:23.164+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:23.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:23.164+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:23.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:23.658+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:23.658+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:23.658+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:23.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:23 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:23.658+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:24.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:24.186+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:24.186+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:24.186+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:24.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:24.186+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:24.252+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:24.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:24.252+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:24.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:24.252+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:24.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:24.252+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:24.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:24.613+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:24.613+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:24.613+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:24.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:24 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:24.613+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:25.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:25.254+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:25.254+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:25.254+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:25.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:25.255+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:25.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:25.153+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:25.153+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:25.153+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:25.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:25.154+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:25.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:25.617+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:25.617+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:25.617+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:25.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:25 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:25.617+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:26.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:26.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:26.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:26.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:26.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:26.245+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:26.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:26.246+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:26.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:26.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:26.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:26.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:26.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:26.180+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:26.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:26.181+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:26.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:26.617+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:26.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:26.617+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:26.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:26.617+0000 7f13742c0640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:39.546026+0000 (oldest deadline 2024-09-20T20:37:59.546026+0000) 2024-09-20T20:38:26.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:26 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0[49355]: 2024-09-20T20:38:26.617+0000 7f13742c0640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:5293 3.6 3:623a8c31:::1000000459d.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-09-20T20:38:27.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:27.224+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:27.224+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:27.224+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:27.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:27.225+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:27.133+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:27.133+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:27.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:27.133+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:27.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:27 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:27.134+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:27.908 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:27 smithi114 podman[88599]: 2024-09-20 20:38:27.616848843 +0000 UTC m=+5.729567999 container died 6f0510eaf505e8b317af997e937307e6401e708267e7c8a06ddf37ff9ab09841 (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0, org.label-schema.vendor=CentOS, RELEASE=squid-ed8ee0f, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-20T20:38:28.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:28 smithi114 podman[88599]: 2024-09-20 20:38:28.005989637 +0000 UTC m=+6.118708790 container cleanup 6f0510eaf505e8b317af997e937307e6401e708267e7c8a06ddf37ff9ab09841 (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-ed8ee0f, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T20:38:28.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:28 smithi114 bash[88599]: ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0 2024-09-20T20:38:28.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:28 smithi114 podman[88615]: 2024-09-20 20:38:28.290864777 +0000 UTC m=+0.672637033 container remove 6f0510eaf505e8b317af997e937307e6401e708267e7c8a06ddf37ff9ab09841 (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-ed8ee0f, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-20T20:38:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:28.197+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:28.197+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:28.197+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:28.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:28.197+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:28.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:28.157+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:28.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:28.157+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:28.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:28.157+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:28.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:28 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:28.157+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:28.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:28 smithi114 podman[88705]: 2024-09-20 20:38:28.525371258 +0000 UTC m=+0.015856490 image pull 6acb0a031eab55fe07acc1e5695be48af529a148c2f67dce7aead6591509af30 quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65 2024-09-20T20:38:28.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:28 smithi114 podman[88705]: 2024-09-20 20:38:28.650154014 +0000 UTC m=+0.140639225 container create aaf3fb78efbc6d8bb4069324a57f0cf4ab12c3f7c7757a9872221a77c408c00d (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=squid-ed8ee0f, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-20T20:38:28.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:28 smithi114 podman[88705]: 2024-09-20 20:38:28.919539675 +0000 UTC m=+0.410024892 container init aaf3fb78efbc6d8bb4069324a57f0cf4ab12c3f7c7757a9872221a77c408c00d (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, RELEASE=squid-ed8ee0f, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-20T20:38:29.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:28 smithi114 podman[88705]: 2024-09-20 20:38:28.922936472 +0000 UTC m=+0.413421695 container start aaf3fb78efbc6d8bb4069324a57f0cf4ab12c3f7c7757a9872221a77c408c00d (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-ed8ee0f, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-20T20:38:29.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:29 smithi114 podman[88705]: 2024-09-20 20:38:29.027859206 +0000 UTC m=+0.518344439 container attach aaf3fb78efbc6d8bb4069324a57f0cf4ab12c3f7c7757a9872221a77c408c00d (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0-deactivate, RELEASE=squid-ed8ee0f, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-20T20:38:29.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:29 smithi114 podman[88705]: 2024-09-20 20:38:29.12089211 +0000 UTC m=+0.611377334 container died aaf3fb78efbc6d8bb4069324a57f0cf4ab12c3f7c7757a9872221a77c408c00d (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-ed8ee0f, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-20T20:38:29.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:29.165+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:29.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:29.165+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:29.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:29.165+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:29.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:29.165+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:29.205+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:29.205+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:29.205+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:29.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:29 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:29.205+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:30.136 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:29 smithi114 systemd[1]: Stopping Ceph osd.1 for 6a49c7bc-7787-11ef-baf5-efdc52797490... 2024-09-20T20:38:30.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:29 smithi114 podman[88727]: 2024-09-20 20:38:29.740416333 +0000 UTC m=+0.613779830 container remove aaf3fb78efbc6d8bb4069324a57f0cf4ab12c3f7c7757a9872221a77c408c00d (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-ed8ee0f, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-20T20:38:30.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:29 smithi114 systemd[1]: ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@osd.0.service: Deactivated successfully. 2024-09-20T20:38:30.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:29 smithi114 systemd[1]: Stopped Ceph osd.0 for 6a49c7bc-7787-11ef-baf5-efdc52797490. 2024-09-20T20:38:30.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 20 20:38:29 smithi114 systemd[1]: ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@osd.0.service: Consumed 27min 14.743s CPU time. 2024-09-20T20:38:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:30.201+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:30.201+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:30.201+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:30.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:30.202+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:30.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:30.133+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:30.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:30.133+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:30.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:30.133+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:30.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:30.134+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:30.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:30.243+0000 7fcc6719a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:38:30.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:30.243+0000 7fcc6719a640 -1 osd.1 46 *** Got signal Terminated *** 2024-09-20T20:38:30.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:30 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:30.243+0000 7fcc6719a640 -1 osd.1 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:38:31.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:31.137+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:31.137+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:31.137+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:31.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:31.138+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:31.208+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:31.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:31.208+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:31.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:31.208+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:31.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:31 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:31.209+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:32.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:32.177+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:32.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:32.177+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:32.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:32.177+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:32.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:32.178+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:32.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:32.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:32.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:32.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:32.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:32.147+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:32.467 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:32 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:32.148+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:33.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:33.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:33.183+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:33.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:33.183+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:33.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:33.165+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:33.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:33.165+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:33.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:33.165+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:33.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:33 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:33.166+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:34.156+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:34.156+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:34.156+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:34.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:34.157+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:34.179+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:34.179+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:34.179+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:34.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:34 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:34.180+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:35.164+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:35.164+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:35.164+0000 7fcc5f797640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:46.961330+0000 (oldest deadline 2024-09-20T20:38:06.961330+0000) 2024-09-20T20:38:35.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1[57031]: 2024-09-20T20:38:35.165+0000 7fcc5f797640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1091 2.9 2:9328e161:::10000004c25.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:35.174+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:35.174+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:35.174+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:35.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:35 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:35.174+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:36.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:35 smithi114 podman[88812]: 2024-09-20 20:38:35.813529135 +0000 UTC m=+5.693532813 container died 98b6e512871748c64f2a8737d48300a942dbcde882feb1183a18076b41ab9111 (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-ed8ee0f, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-20T20:38:36.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:36 smithi114 podman[88812]: 2024-09-20 20:38:36.202750103 +0000 UTC m=+6.082753766 container cleanup 98b6e512871748c64f2a8737d48300a942dbcde882feb1183a18076b41ab9111 (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1, org.label-schema.vendor=CentOS, RELEASE=squid-ed8ee0f, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:38:36.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:36 smithi114 bash[88812]: ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1 2024-09-20T20:38:36.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:36.160+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:36.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:36.160+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:36.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:36.160+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:36.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:36 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:36.161+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:36.825 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:36 smithi114 podman[88826]: 2024-09-20 20:38:36.471127066 +0000 UTC m=+0.653661665 container remove 98b6e512871748c64f2a8737d48300a942dbcde882feb1183a18076b41ab9111 (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=squid-ed8ee0f, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:38:37.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:36 smithi114 podman[88915]: 2024-09-20 20:38:36.726340253 +0000 UTC m=+0.016685908 image pull 6acb0a031eab55fe07acc1e5695be48af529a148c2f67dce7aead6591509af30 quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65 2024-09-20T20:38:37.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:36 smithi114 podman[88915]: 2024-09-20 20:38:36.855492263 +0000 UTC m=+0.145837905 container create 0599958e60818cfd726b5b6f0fecc93bc8079ed66a7c33294f7a0dcdce275954 (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-ed8ee0f, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-20T20:38:37.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:37 smithi114 podman[88915]: 2024-09-20 20:38:37.124687819 +0000 UTC m=+0.415033462 container init 0599958e60818cfd726b5b6f0fecc93bc8079ed66a7c33294f7a0dcdce275954 (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-ed8ee0f, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-20T20:38:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:37 smithi114 podman[88915]: 2024-09-20 20:38:37.128077471 +0000 UTC m=+0.418423121 container start 0599958e60818cfd726b5b6f0fecc93bc8079ed66a7c33294f7a0dcdce275954 (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-ed8ee0f, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-20T20:38:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:37 smithi114 podman[88915]: 2024-09-20 20:38:37.258164123 +0000 UTC m=+0.548509767 container attach 0599958e60818cfd726b5b6f0fecc93bc8079ed66a7c33294f7a0dcdce275954 (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-ed8ee0f, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-20T20:38:37.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:37 smithi114 podman[88915]: 2024-09-20 20:38:37.318042688 +0000 UTC m=+0.608388332 container died 0599958e60818cfd726b5b6f0fecc93bc8079ed66a7c33294f7a0dcdce275954 (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-ed8ee0f, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T20:38:37.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:37.176+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:37.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:37.176+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:37.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:37.176+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:37.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:37 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:37.177+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:38.158+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:38.158+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:38.158+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:38.158+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:38.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:38 smithi114 systemd[1]: Stopping Ceph osd.2 for 6a49c7bc-7787-11ef-baf5-efdc52797490... 2024-09-20T20:38:38.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:37 smithi114 podman[88938]: 2024-09-20 20:38:37.962045953 +0000 UTC m=+0.637550978 container remove 0599958e60818cfd726b5b6f0fecc93bc8079ed66a7c33294f7a0dcdce275954 (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-ed8ee0f, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:38:38.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:37 smithi114 systemd[1]: ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@osd.1.service: Deactivated successfully. 2024-09-20T20:38:38.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:37 smithi114 systemd[1]: Stopped Ceph osd.1 for 6a49c7bc-7787-11ef-baf5-efdc52797490. 2024-09-20T20:38:38.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 20 20:38:37 smithi114 systemd[1]: ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@osd.1.service: Consumed 28min 3.230s CPU time. 2024-09-20T20:38:38.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:38.455+0000 7f7677395640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:38:38.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:38.456+0000 7f7677395640 -1 osd.2 46 *** Got signal Terminated *** 2024-09-20T20:38:38.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:38 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:38.456+0000 7f7677395640 -1 osd.2 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:38:39.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:39.145+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:39.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:39.145+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:39.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:39.145+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:39.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:39 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:39.145+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:40.154+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:40.154+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:40.154+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:40.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:40 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:40.154+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:41.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:41.182+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:41.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:41.182+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:41.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:41.182+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:41.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:41 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:41.183+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:42.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:42.164+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:42.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:42.164+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:42.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:42.164+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:42.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:42 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:42.164+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:43.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:43.183+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6804 osd.3 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:43.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:43.183+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6812 osd.4 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:43.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:43.183+0000 7f767399a640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.160:6820 osd.5 ever on either front or back, first ping sent 2024-09-20T20:37:42.187020+0000 (oldest deadline 2024-09-20T20:38:02.187020+0000) 2024-09-20T20:38:43.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:43 smithi114 ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2[64674]: 2024-09-20T20:38:43.183+0000 7f767399a640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:1097 3.7 3:e2491513:::10000004c5e.00000000:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-20T20:38:44.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:43 smithi114 podman[89023]: 2024-09-20 20:38:43.998557881 +0000 UTC m=+5.666148462 container died dda945a7e7caede83d029ef74b51bdb90c4e23a3ce5cc1c50fd180071f45491d (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-ed8ee0f, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-20T20:38:44.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:44 smithi114 podman[89023]: 2024-09-20 20:38:44.357721781 +0000 UTC m=+6.025312425 container cleanup dda945a7e7caede83d029ef74b51bdb90c4e23a3ce5cc1c50fd180071f45491d (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2, org.label-schema.build-date=20240731, ceph=True, RELEASE=squid-ed8ee0f, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:38:44.618 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:44 smithi114 bash[89023]: ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2 2024-09-20T20:38:44.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:44 smithi114 podman[89039]: 2024-09-20 20:38:44.617228221 +0000 UTC m=+0.613021294 container remove dda945a7e7caede83d029ef74b51bdb90c4e23a3ce5cc1c50fd180071f45491d (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-ed8ee0f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-09-20T20:38:45.213 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:44 smithi114 podman[89128]: 2024-09-20 20:38:44.858037083 +0000 UTC m=+0.015779533 image pull 6acb0a031eab55fe07acc1e5695be48af529a148c2f67dce7aead6591509af30 quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65 2024-09-20T20:38:45.213 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:44 smithi114 podman[89128]: 2024-09-20 20:38:44.96791757 +0000 UTC m=+0.125660005 container create f5ea725fef4b71e340d9361bb45f58802ebe2a749506f8ba013b09548b50c8fa (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-ed8ee0f, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-20T20:38:45.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:45 smithi114 podman[89128]: 2024-09-20 20:38:45.212074968 +0000 UTC m=+0.369817419 container init f5ea725fef4b71e340d9361bb45f58802ebe2a749506f8ba013b09548b50c8fa (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-ed8ee0f, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:38:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:45 smithi114 podman[89128]: 2024-09-20 20:38:45.215302834 +0000 UTC m=+0.373045276 container start f5ea725fef4b71e340d9361bb45f58802ebe2a749506f8ba013b09548b50c8fa (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-ed8ee0f) 2024-09-20T20:38:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:45 smithi114 podman[89128]: 2024-09-20 20:38:45.278888939 +0000 UTC m=+0.436631376 container attach f5ea725fef4b71e340d9361bb45f58802ebe2a749506f8ba013b09548b50c8fa (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-ed8ee0f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T20:38:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:45 smithi114 podman[89128]: 2024-09-20 20:38:45.395650131 +0000 UTC m=+0.553392594 container died f5ea725fef4b71e340d9361bb45f58802ebe2a749506f8ba013b09548b50c8fa (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-ed8ee0f, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-20T20:38:46.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:45 smithi114 podman[89151]: 2024-09-20 20:38:45.932187327 +0000 UTC m=+0.530796516 container remove f5ea725fef4b71e340d9361bb45f58802ebe2a749506f8ba013b09548b50c8fa (image=quay.ceph.io/ceph-ci/ceph@sha256:aa3196c17ef2743d1c74c17c73389e656bc6b43f747b865ccf65c9db3cd9cf65, name=ceph-6a49c7bc-7787-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-ed8ee0f, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-20T20:38:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:45 smithi114 systemd[1]: ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@osd.2.service: Deactivated successfully. 2024-09-20T20:38:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:45 smithi114 systemd[1]: Stopped Ceph osd.2 for 6a49c7bc-7787-11ef-baf5-efdc52797490. 2024-09-20T20:38:46.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 20 20:38:45 smithi114 systemd[1]: ceph-6a49c7bc-7787-11ef-baf5-efdc52797490@osd.2.service: Consumed 27min 46.611s CPU time. 2024-09-20T20:38:52.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:38:52.050 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:38:55.118 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:39:04.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:39:04.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:39:07.183 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:39:19.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:39:19.185 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:39:22.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:39:25.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:39:25.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:39:25.331 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:39:25.331 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-20T20:39:25.331 INFO:tasks.cephadm:Teardown complete 2024-09-20T20:39:25.331 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephadm.py", line 1783, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephadm.py", line 2374, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephadm.py", line 2334, in task with contextutil.nested( File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephadm.py", line 1962, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephadm.py", line 232, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephadm.py", line 416, in _rm_cluster remote.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi160 2024-09-20T20:39:25.350 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-20T20:39:25.363 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/__init__.py", line 641, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-09-20T20:39:25.377 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-20T20:39:25.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-20T20:39:25.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:39:25.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:39:28.463 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:39:37.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:39:37.466 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:39:40.527 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:39:52.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:39:52.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:39:55.602 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:39:58.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:39:58.604 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:39:58.671 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:39:58.671 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-20T20:39:58.722 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/__init__.py", line 641, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/__init__.py", line 637, in task with contextutil.nested( File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/__init__.py", line 221, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi160 2024-09-20T20:39:58.738 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-20T20:39:58.750 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-20T20:39:58.750 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T20:39:58.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:39:58.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:39:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-09-20T20:39:58.778 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T20:39:58.778 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-09-20T20:39:58.779 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv01.front.sepia.ceph.com 3 6 377 14 +459us[ +468us] +/- 48ms 2024-09-20T20:39:58.779 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 17 +194us[ +203us] +/- 59ms 2024-09-20T20:39:58.779 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 78 -1093us[-1084us] +/- 86ms 2024-09-20T20:39:58.779 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T20:40:01.842 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:40:10.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:40:10.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:40:13.903 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:40:25.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:40:25.904 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:40:28.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:40:31.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:40:31.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:40:32.050 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:40:32.051 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-20T20:40:32.051 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi160 2024-09-20T20:40:32.063 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-20T20:40:32.073 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-20T20:40:32.073 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-20T20:40:32.083 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-20T20:40:32.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:40:32.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:40:35.251 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:40:44.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:40:44.252 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:40:47.315 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:40:59.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:40:59.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:41:02.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:41:05.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:41:05.384 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:41:05.455 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:41:05.455 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-20T20:41:05.455 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/selinux.py", line 166, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/selinux.py", line 196, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi160 2024-09-20T20:41:05.462 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-20T20:41:05.473 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-20T20:41:05.483 INFO:teuthology.task.internal:Duration was 4320.322364 seconds 2024-09-20T20:41:05.483 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-20T20:41:05.492 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-20T20:41:05.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-20T20:41:05.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:41:05.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:41:05.542 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T20:41:08.559 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:41:17.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:41:17.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:41:20.623 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:41:32.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:41:32.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:41:35.699 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:41:38.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:41:38.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:41:38.770 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:41:38.771 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-20T20:41:38.771 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi160 2024-09-20T20:41:38.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-20T20:41:38.797 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-20T20:41:38.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-20T20:41:38.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:41:38.833 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:41:41.903 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:41:50.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:41:50.904 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:41:53.967 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:42:05.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:42:05.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:42:09.039 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:42:12.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:42:12.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:42:12.111 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:42:12.111 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-20T20:42:12.111 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi160 2024-09-20T20:42:12.127 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-20T20:42:12.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-20T20:42:12.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:42:12.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:42:12.171 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-09-20T20:42:15.214 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:42:24.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:42:24.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:42:27.283 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:42:39.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:42:39.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:42:42.355 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:42:45.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:42:45.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:42:45.423 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:42:45.423 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-20T20:42:45.423 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi160 2024-09-20T20:42:45.437 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-20T20:42:45.458 INFO:teuthology.task.internal:Transferring archived files... 2024-09-20T20:42:45.458 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi/7912799/remote/smithi114 2024-09-20T20:42:45.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-20T20:42:45.498 DEBUG:teuthology.misc:Transferring archived files from smithi160:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi/7912799/remote/smithi160 2024-09-20T20:42:45.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:42:45.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:42:48.563 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:42:57.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:42:57.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:43:00.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:43:12.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:43:12.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:43:15.698 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:43:18.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:43:18.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:43:18.767 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:43:18.769 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-20T20:43:18.770 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/misc.py", line 735, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi160 2024-09-20T20:43:18.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-20T20:43:18.800 INFO:teuthology.task.internal:Not uploading archives. 2024-09-20T20:43:18.800 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-20T20:43:18.861 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-20T20:43:18.862 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-20T20:43:18.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:43:18.869 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:43:18.884 INFO:teuthology.orchestra.run.smithi114.stdout: 265324 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 20 20:39 /home/ubuntu/cephtest 2024-09-20T20:43:18.885 INFO:teuthology.orchestra.run.smithi114.stdout: 267136 768 -rwxr-xr-x 1 ubuntu ubuntu 784376 Sep 20 19:34 /home/ubuntu/cephtest/cephadm 2024-09-20T20:43:18.885 INFO:teuthology.orchestra.run.smithi114.stdout: 265325 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Sep 20 20:42 /home/ubuntu/cephtest/archive 2024-09-20T20:43:18.885 INFO:teuthology.orchestra.run.smithi114.stdout: 265330 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 20 19:29 /home/ubuntu/cephtest/archive/syslog 2024-09-20T20:43:18.885 INFO:teuthology.orchestra.run.smithi114.stdout: 265331 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Sep 20 19:29 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T20:43:18.885 INFO:teuthology.orchestra.run.smithi114.stdout: 265332 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Sep 20 19:29 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T20:43:18.885 INFO:teuthology.orchestra.run.smithi114.stdout: 396700 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 20 20:40 /home/ubuntu/cephtest/archive/audit 2024-09-20T20:43:18.885 INFO:teuthology.orchestra.run.smithi114.stdout: 426278 184 -rw------- 1 ubuntu ubuntu 188231 Sep 20 20:40 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-09-20T20:43:18.886 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-20T20:43:21.935 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:43:30.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:43:30.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:43:33.999 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:43:45.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:43:46.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:43:49.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:43:52.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-20T20:43:52.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:43:52.143 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-20T20:43:52.143 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-20T20:43:52.143 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi160 2024-09-20T20:43:52.158 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-20T20:43:52.172 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-20T20:43:52.328 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-09-20T20:43:52.331 INFO:teuthology.run:Summary data: description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 4320.322364330292 failure_reason: 'Command failed on smithi114 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a49c7bc-7787-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_adking@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a9bc070680ac4a29b9541a5b9826fc8d status: fail success: false 2024-09-20T20:43:52.332 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-20T20:43:52.420 INFO:teuthology.run:FAIL